Commit 56044cac92cdd65dc3b4fd03557eaf32976e6da9 - matrix-synapse

+9

-6

.circleci/config.yml less more

4	4	- image: docker:git
5	5	steps:
6	6	- checkout
7		- setup_remote_docker
8	7	- docker_prepare
9	8	- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
	9	# for release builds, we want to get the amd64 image out asap, so first
	10	# we do an amd64-only build, before following up with a multiarch build.
10	11	- docker_build:
11	12	tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
12	13	platforms: linux/amd64

19	20	- image: docker:git
20	21	steps:
21	22	- checkout
22		- setup_remote_docker
23	23	- docker_prepare
24	24	- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
25		- docker_build:
26		tag: -t matrixdotorg/synapse:latest
27		platforms: linux/amd64
	25	# for `latest`, we don't want the arm images to disappear, so don't update the tag
	26	# until all of the platforms are built.
28	27	- docker_build:
29	28	tag: -t matrixdotorg/synapse:latest
30	29	platforms: linux/amd64,linux/arm/v7,linux/arm64

45	44
46	45	commands:
47	46	docker_prepare:
48		description: Downloads the buildx cli plugin and enables multiarch images
	47	description: Sets up a remote docker server, downloads the buildx cli plugin, and enables multiarch images
49	48	parameters:
50	49	buildx_version:
51	50	type: string
52	51	default: "v0.4.1"
53	52	steps:
	53	- setup_remote_docker:
	54	# 19.03.13 was the most recent available on circleci at the time of
	55	# writing.
	56	version: 19.03.13
54	57	- run: apk add --no-cache curl
55	58	- run: mkdir -vp ~/.docker/cli-plugins/ ~/dockercache
56	59	- run: curl --silent -L "https://github.com/docker/buildx/releases/download/<< parameters.buildx_version >>/buildx-<< parameters.buildx_version >>.linux-amd64" > ~/.docker/cli-plugins/docker-buildx

+187

-0

CHANGES.md less more

	0	Synapse 1.25.0 (2021-01-13)
	1	===========================
	2
	3	Ending Support for Python 3.5 and Postgres 9.5
	4	----------------------------------------------
	5
	6	With this release, the Synapse team is announcing a formal deprecation policy for our platform dependencies, like Python and PostgreSQL:
	7
	8	All future releases of Synapse will follow the upstream end-of-life schedules.
	9
	10	Which means:
	11
	12	* This is the last release which guarantees support for Python 3.5.
	13	* We will end support for PostgreSQL 9.5 early next month.
	14	* We will end support for Python 3.6 and PostgreSQL 9.6 near the end of the year.
	15
	16	Crucially, this means __we will not produce .deb packages for Debian 9 (Stretch) or Ubuntu 16.04 (Xenial)__ beyond the transition period described below.
	17
	18	The website https://endoflife.date/ has convenient summaries of the support schedules for projects like [Python](https://endoflife.date/python) and [PostgreSQL](https://endoflife.date/postgresql).
	19
	20	If you are unable to upgrade your environment to a supported version of Python or Postgres, we encourage you to consider using the [Synapse Docker images](./INSTALL.md#docker-images-and-ansible-playbooks) instead.
	21
	22	### Transition Period
	23
	24	We will make a good faith attempt to avoid breaking compatibility in all releases through the end of March 2021. However, critical security vulnerabilities in dependencies or other unanticipated circumstances may arise which necessitate breaking compatibility earlier.
	25
	26	We intend to continue producing .deb packages for Debian 9 (Stretch) and Ubuntu 16.04 (Xenial) through the transition period.
	27
	28	Removal warning
	29	---------------
	30
	31	The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md)
	32	and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md)
	33	are deprecated and will be removed in a future release. They will be replaced by the
	34	[Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api).
	35
	36	`POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and
	37	`POST /_synapse/admin/v1/shutdown_room/<room_id>`.
	38
	39	Bugfixes
	40	--------
	41
	42	- Fix HTTP proxy support when using a proxy that is on a blacklisted IP. Introduced in v1.25.0rc1. Contributed by @Bubu. ([\#9084](https://github.com/matrix-org/synapse/issues/9084))
	43
	44
	45	Synapse 1.25.0rc1 (2021-01-06)
	46	==============================
	47
	48	Features
	49	--------
	50
	51	- Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756))
	52	- Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853))
	53	- Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874))
	54	- Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886))
	55	- Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887))
	56	- Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890))
	57	- Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951))
	58	- Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930))
	59	- The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931))
	60	- Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954))
	61	- Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970))
	62	- Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994))
	63
	64
	65	Bugfixes
	66	--------
	67
	68	- Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827))
	69	- Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837))
	70	- Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858))
	71	- Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862))
	72	- Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865))
	73	- Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867))
	74	- Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872))
	75	- Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883))
	76	- Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918))
	77	- Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920))
	78	- Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921))
	79	- Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964))
	80	- Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937))
	81	- Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945))
	82	- Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959))
	83	- Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962))
	84	- Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965))
	85	- Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971))
	86	- Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975))
	87	- Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977))
	88
	89
	90	Improved Documentation
	91	----------------------
	92
	93	- Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802))
	94	- Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839))
	95	- Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873))
	96	- Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891))
	97	- Move instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by @fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987))
	98	- Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992))
	99	- Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002))
	100
	101
	102	Deprecations and Removals
	103	-------------------------
	104
	105	- Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829))
	106
	107
	108	Internal Changes
	109	----------------
	110
	111	- Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958))
	112	- Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861))
	113	- Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864))
	114	- Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879))
	115	- Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020))
	116	- Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881))
	117	- Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905))
	118	- Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906))
	119	- Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909))
	120	- Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935))
	121	- Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929))
	122	- Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946))
	123	- Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950))
	124	- Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952))
	125	- Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963))
	126	- Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973))
	127	- Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976))
	128	- Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979))
	129	- Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980))
	130	- Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986))
	131	- Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998))
	132	- Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))
	133
	134
0	135	Synapse 1.24.0 (2020-12-09)
1	136	===========================
2	137

36	171	```
37	172	* Administrators who have installed Synapse from distribution packages should
38	173	consult the information from their distributions.
	174
	175	Internal Changes
	176	----------------
	177
	178	- Add a maximum version for pysaml2 on Python 3.5. ([\#8898](https://github.com/matrix-org/synapse/issues/8898))
	179
	180
	181	Synapse 1.23.1 (2020-12-09)
	182	===========================
	183
	184	Due to the two security issues highlighted below, server administrators are
	185	encouraged to update Synapse. We are not aware of these vulnerabilities being
	186	exploited in the wild.
	187
	188	Security advisory
	189	-----------------
	190
	191	The following issues are fixed in v1.23.1 and v1.24.0.
	192
	193	- There is a denial of service attack
	194	([CVE-2020-26257](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-26257))
	195	against the federation APIs in which future events will not be correctly sent
	196	to other servers over federation. This affects all servers that participate in
	197	open federation. (Fixed in [#8776](https://github.com/matrix-org/synapse/pull/8776)).
	198
	199	- Synapse may be affected by OpenSSL
	200	[CVE-2020-1971](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1971).
	201	Synapse administrators should ensure that they have the latest versions of
	202	the cryptography Python package installed.
	203
	204	To upgrade Synapse along with the cryptography package:
	205
	206	* Administrators using the [`matrix.org` Docker
	207	image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu
	208	packages from
	209	`matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages)
	210	should ensure that they have version 1.24.0 or 1.23.1 installed: these images include
	211	the updated packages.
	212	* Administrators who have [installed Synapse from
	213	source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source)
	214	should upgrade the cryptography package within their virtualenv by running:
	215	```sh
	216	<path_to_virtualenv>/bin/pip install 'cryptography>=3.3'
	217	```
	218	* Administrators who have installed Synapse from distribution packages should
	219	consult the information from their distributions.
	220
	221	Bugfixes
	222	--------
	223
	224	- Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body. ([\#8776](https://github.com/matrix-org/synapse/issues/8776))
	225
39	226
40	227	Internal Changes
41	228	----------------

+153

-135

INSTALL.md less more

0		- [Choosing your server name](#choosing-your-server-name)
1		- [Picking a database engine](#picking-a-database-engine)
2		- [Installing Synapse](#installing-synapse)
3		- [Installing from source](#installing-from-source)
4		- [Platform-Specific Instructions](#platform-specific-instructions)
5		- [Prebuilt packages](#prebuilt-packages)
6		- [Setting up Synapse](#setting-up-synapse)
7		- [TLS certificates](#tls-certificates)
8		- [Client Well-Known URI](#client-well-known-uri)
9		- [Email](#email)
10		- [Registering a user](#registering-a-user)
11		- [Setting up a TURN server](#setting-up-a-turn-server)
12		- [URL previews](#url-previews)
13		- [Troubleshooting Installation](#troubleshooting-installation)
14
15		# Choosing your server name
	0	# Installation Instructions
	1
	2	There are 3 steps to follow under Installation Instructions.
	3
	4	- [Installation Instructions](#installation-instructions)
	5	- [Choosing your server name](#choosing-your-server-name)
	6	- [Installing Synapse](#installing-synapse)
	7	- [Installing from source](#installing-from-source)
	8	- [Platform-Specific Instructions](#platform-specific-instructions)
	9	- [Debian/Ubuntu/Raspbian](#debianubunturaspbian)
	10	- [ArchLinux](#archlinux)
	11	- [CentOS/Fedora](#centosfedora)
	12	- [macOS](#macos)
	13	- [OpenSUSE](#opensuse)
	14	- [OpenBSD](#openbsd)
	15	- [Windows](#windows)
	16	- [Prebuilt packages](#prebuilt-packages)
	17	- [Docker images and Ansible playbooks](#docker-images-and-ansible-playbooks)
	18	- [Debian/Ubuntu](#debianubuntu)
	19	- [Matrix.org packages](#matrixorg-packages)
	20	- [Downstream Debian packages](#downstream-debian-packages)
	21	- [Downstream Ubuntu packages](#downstream-ubuntu-packages)
	22	- [Fedora](#fedora)
	23	- [OpenSUSE](#opensuse-1)
	24	- [SUSE Linux Enterprise Server](#suse-linux-enterprise-server)
	25	- [ArchLinux](#archlinux-1)
	26	- [Void Linux](#void-linux)
	27	- [FreeBSD](#freebsd)
	28	- [OpenBSD](#openbsd-1)
	29	- [NixOS](#nixos)
	30	- [Setting up Synapse](#setting-up-synapse)
	31	- [Using PostgreSQL](#using-postgresql)
	32	- [TLS certificates](#tls-certificates)
	33	- [Client Well-Known URI](#client-well-known-uri)
	34	- [Email](#email)
	35	- [Registering a user](#registering-a-user)
	36	- [Setting up a TURN server](#setting-up-a-turn-server)
	37	- [URL previews](#url-previews)
	38	- [Troubleshooting Installation](#troubleshooting-installation)
	39
	40	## Choosing your server name
16	41
17	42	It is important to choose the name for your server before you install Synapse,
18	43	because it cannot be changed later.

28	53	`user@email.example.com`) - but doing so may require more advanced setup: see
29	54	[Setting up Federation](docs/federate.md).
30	55
31		# Picking a database engine
32
33		Synapse offers two database engines:
34		* [PostgreSQL](https://www.postgresql.org)
35		* [SQLite](https://sqlite.org/)
36
37		Almost all installations should opt to use PostgreSQL. Advantages include:
38
39		* significant performance improvements due to the superior threading and
40		caching model, smarter query optimiser
41		* allowing the DB to be run on separate hardware
42
43		For information on how to install and use PostgreSQL, please see
44		[docs/postgres.md](docs/postgres.md)
45
46		By default Synapse uses SQLite and in doing so trades performance for convenience.
47		SQLite is only recommended in Synapse for testing purposes or for servers with
48		light workloads.
49
50		# Installing Synapse
51
52		## Installing from source
	56	## Installing Synapse
	57
	58	### Installing from source
53	59
54	60	(Prebuilt packages are available for some platforms - see [Prebuilt packages](#prebuilt-packages).)
55	61

67	73
68	74	To install the Synapse homeserver run:
69	75
70		```
	76	```sh
71	77	mkdir -p ~/synapse
72	78	virtualenv -p python3 ~/synapse/env
73	79	source ~/synapse/env/bin/activate

84	90	This Synapse installation can then be later upgraded by using pip again with the
85	91	update flag:
86	92
87		```
	93	```sh
88	94	source ~/synapse/env/bin/activate
89	95	pip install -U matrix-synapse
90	96	```

92	98	Before you can start Synapse, you will need to generate a configuration
93	99	file. To do this, run (in your virtualenv, as before):
94	100
95		```
	101	```sh
96	102	cd ~/synapse
97	103	python -m synapse.app.homeserver \
98	104	--server-name my.domain.name \

110	116	change your homeserver's keys, you may find that other homeserver have the
111	117	old key cached. If you update the signing key, you should change the name of the
112	118	key in the `<server name>.signing.key` file (the second word) to something
113		different. See the
114		[spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
115		for more information on key management).
	119	different. See the [spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys) for more information on key management).
116	120
117	121	To actually run your new homeserver, pick a working directory for Synapse to
118	122	run (e.g. `~/synapse`), and:
119	123
120		```
	124	```sh
121	125	cd ~/synapse
122	126	source env/bin/activate
123	127	synctl start
124	128	```
125	129
126		### Platform-Specific Instructions
127
128		#### Debian/Ubuntu/Raspbian
	130	#### Platform-Specific Instructions
	131
	132	##### Debian/Ubuntu/Raspbian
129	133
130	134	Installing prerequisites on Ubuntu or Debian:
131	135
132		```
133		sudo apt-get install build-essential python3-dev libffi-dev \
	136	```sh
	137	sudo apt install build-essential python3-dev libffi-dev \
134	138	python3-pip python3-setuptools sqlite3 \
135	139	libssl-dev virtualenv libjpeg-dev libxslt1-dev
136	140	```
137	141
138		#### ArchLinux
	142	##### ArchLinux
139	143
140	144	Installing prerequisites on ArchLinux:
141	145
142		```
	146	```sh
143	147	sudo pacman -S base-devel python python-pip \
144	148	python-setuptools python-virtualenv sqlite3
145	149	```
146	150
147		#### CentOS/Fedora
	151	##### CentOS/Fedora
148	152
149	153	Installing prerequisites on CentOS 8 or Fedora>26:
150	154
151		```
	155	```sh
152	156	sudo dnf install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
153	157	libwebp-devel tk-devel redhat-rpm-config \
154	158	python3-virtualenv libffi-devel openssl-devel

157	161
158	162	Installing prerequisites on CentOS 7 or Fedora<=25:
159	163
160		```
	164	```sh
161	165	sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
162	166	lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
163	167	python3-virtualenv libffi-devel openssl-devel

169	173	recent SQLite version, but it is recommended that you instead use a Postgres
170	174	database: see [docs/postgres.md](docs/postgres.md).
171	175
172		#### macOS
	176	##### macOS
173	177
174	178	Installing prerequisites on macOS:
175	179
176		```
	180	```sh
177	181	xcode-select --install
178	182	sudo easy_install pip
179	183	sudo pip install virtualenv

183	187	On macOS Catalina (10.15) you may need to explicitly install OpenSSL
184	188	via brew and inform `pip` about it so that `psycopg2` builds:
185	189
186		```
	190	```sh
187	191	brew install openssl@1.1
188	192	export LDFLAGS=-L/usr/local/Cellar/openssl\@1.1/1.1.1d/lib/
189	193	```
190	194
191		#### OpenSUSE
	195	##### OpenSUSE
192	196
193	197	Installing prerequisites on openSUSE:
194	198
195		```
	199	```sh
196	200	sudo zypper in -t pattern devel_basis
197	201	sudo zypper in python-pip python-setuptools sqlite3 python-virtualenv \
198	202	python-devel libffi-devel libopenssl-devel libjpeg62-devel
199	203	```
200	204
201		#### OpenBSD
	205	##### OpenBSD
202	206
203	207	A port of Synapse is available under `net/synapse`. The filesystem
204	208	underlying the homeserver directory (defaults to `/var/synapse`) has to be

212	216	Creating a `WRKOBJDIR` for building python under `/usr/local` (which on a
213	217	default OpenBSD installation is mounted with `wxallowed`):
214	218
215		```
	219	```sh
216	220	doas mkdir /usr/local/pobj_wxallowed
217	221	```
218	222
219	223	Assuming `PORTS_PRIVSEP=Yes` (cf. `bsd.port.mk(5)`) and `SUDO=doas` are
220	224	configured in `/etc/mk.conf`:
221	225
222		```
	226	```sh
223	227	doas chown _pbuild:_pbuild /usr/local/pobj_wxallowed
224	228	```
225	229
226	230	Setting the `WRKOBJDIR` for building python:
227	231
228		```
	232	```sh
229	233	echo WRKOBJDIR_lang/python/3.7=/usr/local/pobj_wxallowed \\nWRKOBJDIR_lang/python/2.7=/usr/local/pobj_wxallowed >> /etc/mk.conf
230	234	```
231	235
232	236	Building Synapse:
233	237
234		```
	238	```sh
235	239	cd /usr/ports/net/synapse
236	240	make install
237	241	```
238	242
239		#### Windows
	243	##### Windows
240	244
241	245	If you wish to run or develop Synapse on Windows, the Windows Subsystem For
242	246	Linux provides a Linux environment on Windows 10 which is capable of using the
243	247	Debian, Fedora, or source installation methods. More information about WSL can
244		be found at https://docs.microsoft.com/en-us/windows/wsl/install-win10 for
245		Windows 10 and https://docs.microsoft.com/en-us/windows/wsl/install-on-server
	248	be found at <https://docs.microsoft.com/en-us/windows/wsl/install-win10> for
	249	Windows 10 and <https://docs.microsoft.com/en-us/windows/wsl/install-on-server>
246	250	for Windows Server.
247	251
248		## Prebuilt packages
	252	### Prebuilt packages
249	253
250	254	As an alternative to installing from source, prebuilt packages are available
251	255	for a number of platforms.
252	256
253		### Docker images and Ansible playbooks
	257	#### Docker images and Ansible playbooks
254	258
255	259	There is an offical synapse image available at
256		https://hub.docker.com/r/matrixdotorg/synapse which can be used with
	260	<https://hub.docker.com/r/matrixdotorg/synapse> which can be used with
257	261	the docker-compose file available at [contrib/docker](contrib/docker). Further
258	262	information on this including configuration options is available in the README
259	263	on hub.docker.com.
260	264
261	265	Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
262	266	Dockerfile to automate a synapse server in a single Docker image, at
263		https://hub.docker.com/r/avhost/docker-matrix/tags/
	267	<https://hub.docker.com/r/avhost/docker-matrix/tags/>
264	268
265	269	Slavi Pantaleev has created an Ansible playbook,
266	270	which installs the offical Docker image of Matrix Synapse
267	271	along with many other Matrix-related services (Postgres database, Element, coturn,
268	272	ma1sd, SSL support, etc.).
269	273	For more details, see
270		https://github.com/spantaleev/matrix-docker-ansible-deploy
271
272
273		### Debian/Ubuntu
274
275		#### Matrix.org packages
	274	<https://github.com/spantaleev/matrix-docker-ansible-deploy>
	275
	276	#### Debian/Ubuntu
	277
	278	##### Matrix.org packages
276	279
277	280	Matrix.org provides Debian/Ubuntu packages of the latest stable version of
278		Synapse via https://packages.matrix.org/debian/. They are available for Debian
	281	Synapse via <https://packages.matrix.org/debian/>. They are available for Debian
279	282	9 (Stretch), Ubuntu 16.04 (Xenial), and later. To use them:
280	283
281		```
	284	```sh
282	285	sudo apt install -y lsb-release wget apt-transport-https
283	286	sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
284	287	echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" \|

298	301	/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
299	302	`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
300	303
301		#### Downstream Debian packages
	304	##### Downstream Debian packages
302	305
303	306	We do not recommend using the packages from the default Debian `buster`
304	307	repository at this time, as they are old and suffer from known security

310	313	If you are using Debian `sid` or testing, Synapse is available in the default
311	314	repositories and it should be possible to install it simply with:
312	315
313		```
	316	```sh
314	317	sudo apt install matrix-synapse
315	318	```
316	319
317		#### Downstream Ubuntu packages
	320	##### Downstream Ubuntu packages
318	321
319	322	We do not recommend using the packages in the default Ubuntu repository
320	323	at this time, as they are old and suffer from known security vulnerabilities.
321	324	The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
322	325
323		### Fedora
	326	#### Fedora
324	327
325	328	Synapse is in the Fedora repositories as `matrix-synapse`:
326	329
327		```
	330	```sh
328	331	sudo dnf install matrix-synapse
329	332	```
330	333
331	334	Oleg Girko provides Fedora RPMs at
332		https://obs.infoserver.lv/project/monitor/matrix-synapse
333
334		### OpenSUSE
	335	<https://obs.infoserver.lv/project/monitor/matrix-synapse>
	336
	337	#### OpenSUSE
335	338
336	339	Synapse is in the OpenSUSE repositories as `matrix-synapse`:
337	340
338		```
	341	```sh
339	342	sudo zypper install matrix-synapse
340	343	```
341	344
342		### SUSE Linux Enterprise Server
	345	#### SUSE Linux Enterprise Server
343	346
344	347	Unofficial package are built for SLES 15 in the openSUSE:Backports:SLE-15 repository at
345		https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/
346
347		### ArchLinux
	348	<https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/>
	349
	350	#### ArchLinux
348	351
349	352	The quickest way to get up and running with ArchLinux is probably with the community package
350		https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
	353	<https://www.archlinux.org/packages/community/any/matrix-synapse/>, which should pull in most of
351	354	the necessary dependencies.
352	355
353	356	pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 ):
354	357
355		```
	358	```sh
356	359	sudo pip install --upgrade pip
357	360	```
358	361

361	364	compile it under the right architecture. (This should not be needed if
362	365	installing under virtualenv):
363	366
364		```
	367	```sh
365	368	sudo pip uninstall py-bcrypt
366	369	sudo pip install py-bcrypt
367	370	```
368	371
369		### Void Linux
	372	#### Void Linux
370	373
371	374	Synapse can be found in the void repositories as 'synapse':
372	375
373		```
	376	```sh
374	377	xbps-install -Su
375	378	xbps-install -S synapse
376	379	```
377	380
378		### FreeBSD
	381	#### FreeBSD
379	382
380	383	Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
381	384
382		- Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
383		- Packages: `pkg install py37-matrix-synapse`
384
385		### OpenBSD
	385	- Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
	386	- Packages: `pkg install py37-matrix-synapse`
	387
	388	#### OpenBSD
386	389
387	390	As of OpenBSD 6.7 Synapse is available as a pre-compiled binary. The filesystem
388	391	underlying the homeserver directory (defaults to `/var/synapse`) has to be

391	394
392	395	Installing Synapse:
393	396
394		```
	397	```sh
395	398	doas pkg_add synapse
396	399	```
397	400
398		### NixOS
	401	#### NixOS
399	402
400	403	Robin Lambertz has packaged Synapse for NixOS at:
401		https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix
402
403		# Setting up Synapse
	404	<https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix>
	405
	406	## Setting up Synapse
404	407
405	408	Once you have installed synapse as above, you will need to configure it.
406	409
407		## TLS certificates
	410	### Using PostgreSQL
	411
	412	By default Synapse uses [SQLite](https://sqlite.org/) and in doing so trades performance for convenience.
	413	SQLite is only recommended in Synapse for testing purposes or for servers with
	414	very light workloads.
	415
	416	Almost all installations should opt to use [PostgreSQL](https://www.postgresql.org). Advantages include:
	417
	418	- significant performance improvements due to the superior threading and
	419	caching model, smarter query optimiser
	420	- allowing the DB to be run on separate hardware
	421
	422	For information on how to install and use PostgreSQL in Synapse, please see
	423	[docs/postgres.md](docs/postgres.md)
	424
	425	### TLS certificates
408	426
409	427	The default configuration exposes a single HTTP port on the local
410	428	interface: `http://localhost:8008`. It is suitable for local testing,

418	436	Alternatively, you can configure Synapse to expose an HTTPS port. To do
419	437	so, you will need to edit `homeserver.yaml`, as follows:
420	438
421		* First, under the `listeners` section, uncomment the configuration for the
	439	- First, under the `listeners` section, uncomment the configuration for the
422	440	TLS-enabled listener. (Remove the hash sign (`#`) at the start of
423	441	each line). The relevant lines are like this:
424	442
	443	```yaml
	444	- port: 8448
	445	type: http
	446	tls: true
	447	resources:
	448	- names: [client, federation]
425	449	```
426		- port: 8448
427		type: http
428		tls: true
429		resources:
430		- names: [client, federation]
431		```
432
433		* You will also need to uncomment the `tls_certificate_path` and
	450
	451	- You will also need to uncomment the `tls_certificate_path` and
434	452	`tls_private_key_path` lines under the `TLS` section. You will need to manage
435	453	provisioning of these certificates yourself — Synapse had built-in ACME
436	454	support, but the ACMEv1 protocol Synapse implements is deprecated, not

445	463	For a more detailed guide to configuring your server for federation, see
446	464	[federate.md](docs/federate.md).
447	465
448		## Client Well-Known URI
	466	### Client Well-Known URI
449	467
450	468	Setting up the client Well-Known URI is optional but if you set it up, it will
451	469	allow users to enter their full username (e.g. `@user:<server_name>`) into clients

456	474	The URL `https://<server_name>/.well-known/matrix/client` should return JSON in
457	475	the following format.
458	476
459		```
	477	```json
460	478	{
461	479	"m.homeserver": {
462	480	"base_url": "https://<matrix.example.com>"

466	484
467	485	It can optionally contain identity server information as well.
468	486
469		```
	487	```json
470	488	{
471	489	"m.homeserver": {
472	490	"base_url": "https://<matrix.example.com>"

483	501	view it.
484	502
485	503	In nginx this would be something like:
486		```
	504
	505	```nginx
487	506	location /.well-known/matrix/client {
488	507	return 200 '{"m.homeserver": {"base_url": "https://<matrix.example.com>"}}';
489	508	default_type application/json;

496	515	connect to your server. This is the same URL you put for the `m.homeserver`
497	516	`base_url` above.
498	517
499		```
	518	```yaml
500	519	public_baseurl: "https://<matrix.example.com>"
501	520	```
502	521
503		## Email
	522	### Email
504	523
505	524	It is desirable for Synapse to have the capability to send email. This allows
506	525	Synapse to send password reset emails, send verifications when an email address

515	534	If email is not configured, password reset, registration and notifications via
516	535	email will be disabled.
517	536
518		## Registering a user
	537	### Registering a user
519	538
520	539	The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
521	540

523	542
524	543	This can be done as follows:
525	544
526		```
	545	```sh
527	546	$ source ~/synapse/env/bin/activate
528	547	$ synctl start # if not already running
529	548	$ register_new_matrix_user -c homeserver.yaml http://localhost:8008

541	560	anyone with knowledge of it can register users, including admin accounts,
542	561	on your server even if `enable_registration` is `false`.
543	562
544		## Setting up a TURN server
	563	### Setting up a TURN server
545	564
546	565	For reliable VoIP calls to be routed via this homeserver, you MUST configure
547	566	a TURN server. See [docs/turn-howto.md](docs/turn-howto.md) for details.
548	567
549		## URL previews
	568	### URL previews
550	569
551	570	Synapse includes support for previewing URLs, which is disabled by default. To
552	571	turn it on you must enable the `url_preview_enabled: True` config parameter

556	575	spidering 'internal' URLs on your network. At the very least we recommend that
557	576	your loopback and RFC1918 IP addresses are blacklisted.
558	577
559		This also requires the optional `lxml` and `netaddr` python dependencies to be
560		installed. This in turn requires the `libxml2` library to be available - on
561		Debian/Ubuntu this means `apt-get install libxml2-dev`, or equivalent for
562		your OS.
563
564		# Troubleshooting Installation
	578	This also requires the optional `lxml` python dependency to be installed. This
	579	in turn requires the `libxml2` library to be available - on Debian/Ubuntu this
	580	means `apt-get install libxml2-dev`, or equivalent for your OS.
	581
	582	### Troubleshooting Installation
565	583
566	584	`pip` seems to leak lots of memory during installation. For instance, a Linux
567	585	host with 512MB of RAM may run out of memory whilst installing Twisted. If this
568	586	happens, you will have to individually install the dependencies which are
569	587	failing, e.g.:
570	588
571		```
	589	```sh
572	590	pip install twisted
573	591	```
574	592

+2

-0

README.rst less more

242	242	Synapse Development
243	243	===================
244	244
	245	Join our developer community on Matrix: [#synapse-dev:matrix.org](https://matrix.to/#/#synapse-dev:matrix.org)
	246
245	247	Before setting up a development environment for synapse, make sure you have the
246	248	system dependencies (such as the python header files) installed - see
247	249	`Installing from source <INSTALL.md#installing-from-source>`_.

+43

-0

UPGRADE.rst less more

4	4	version you currently have installed to the current version of Synapse. The extra
5	5	instructions that may be required are listed later in this document.
6	6
	7	* Check that your versions of Python and PostgreSQL are still supported.
	8
	9	Synapse follows upstream lifecycles for `Python`_ and `PostgreSQL`_, and
	10	removes support for versions which are no longer maintained.
	11
	12	The website https://endoflife.date also offers convenient summaries.
	13
	14	.. _Python: https://devguide.python.org/devcycle/#end-of-life-branches
	15	.. _PostgreSQL: https://www.postgresql.org/support/versioning/
	16
7	17	* If Synapse was installed using `prebuilt packages
8	18	<INSTALL.md#prebuilt-packages>`_, you will need to follow the normal process
9	19	for upgrading those packages.

73	83	# replace `1.3.0` and `stretch` accordingly:
74	84	wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
75	85	dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
	86
	87	Upgrading to v1.25.0
	88	====================
	89
	90	Last release supporting Python 3.5
	91	----------------------------------
	92
	93	This is the last release of Synapse which guarantees support with Python 3.5,
	94	which passed its upstream End of Life date several months ago.
	95
	96	We will attempt to maintain support through March 2021, but without guarantees.
	97
	98	In the future, Synapse will follow upstream schedules for ending support of
	99	older versions of Python and PostgreSQL. Please upgrade to at least Python 3.6
	100	and PostgreSQL 9.6 as soon as possible.
	101
	102	Blacklisting IP ranges
	103	----------------------
	104
	105	Synapse v1.25.0 includes new settings, ``ip_range_blacklist`` and
	106	``ip_range_whitelist``, for controlling outgoing requests from Synapse for federation,
	107	identity servers, push, and for checking key validity for third-party invite events.
	108	The previous setting, ``federation_ip_range_blacklist``, is deprecated. The new
	109	``ip_range_blacklist`` defaults to private IP ranges if it is not defined.
	110
	111	If you have never customised ``federation_ip_range_blacklist`` it is recommended
	112	that you remove that setting.
	113
	114	If you have customised ``federation_ip_range_blacklist`` you should update the
	115	setting name to ``ip_range_blacklist``.
	116
	117	If you have a custom push server that is reached via private IP space you may
	118	need to customise ``ip_range_blacklist`` or ``ip_range_whitelist``.
76	119
77	120	Upgrading to v1.24.0
78	121	====================

+18

-0

contrib/prometheus/synapse-v2.rules less more

57	57	labels:
58	58	type: "PDU"
59	59	expr: 'synapse_federation_transaction_queue_pending_pdus + 0'
	60
	61	- record: synapse_storage_events_persisted_by_source_type
	62	expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_type="remote"})
	63	labels:
	64	type: remote
	65	- record: synapse_storage_events_persisted_by_source_type
	66	expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity="client",origin_type="local"})
	67	labels:
	68	type: local
	69	- record: synapse_storage_events_persisted_by_source_type
	70	expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity!="client",origin_type="local"})
	71	labels:
	72	type: bridges
	73	- record: synapse_storage_events_persisted_by_event_type
	74	expr: sum without(origin_entity, origin_type) (synapse_storage_events_persisted_events_sep)
	75	- record: synapse_storage_events_persisted_by_origin
	76	expr: sum without(type) (synapse_storage_events_persisted_events_sep)
	77

+17

-0

debian/changelog less more

	0	matrix-synapse-py3 (1.25.0) stable; urgency=medium
	1
	2	[ Dan Callahan ]
	3	* Update dependencies to account for the removal of the transitional
	4	dh-systemd package from Debian Bullseye.
	5
	6	[ Synapse Packaging team ]
	7	* New synapse release 1.25.0.
	8
	9	-- Synapse Packaging team <packages@matrix.org> Wed, 13 Jan 2021 10:14:55 +0000
	10
0	11	matrix-synapse-py3 (1.24.0) stable; urgency=medium
1	12
2	13	* New synapse release 1.24.0.
3	14
4	15	-- Synapse Packaging team <packages@matrix.org> Wed, 09 Dec 2020 10:14:30 +0000
	16
	17	matrix-synapse-py3 (1.23.1) stable; urgency=medium
	18
	19	* New synapse release 1.23.1.
	20
	21	-- Synapse Packaging team <packages@matrix.org> Wed, 09 Dec 2020 10:40:39 +0000
5	22
6	23	matrix-synapse-py3 (1.23.0) stable; urgency=medium
7	24

+4

-2

debian/control less more

2	2	Priority: extra
3	3	Maintainer: Synapse Packaging team <packages@matrix.org>
4	4	# keep this list in sync with the build dependencies in docker/Dockerfile-dhvirtualenv.
	5	# TODO: Remove the dependency on dh-systemd after dropping support for Ubuntu xenial
	6	# On all other supported releases, it's merely a transitional package which
	7	# does nothing but depends on debhelper (> 9.20160709)
5	8	Build-Depends:
6		debhelper (>= 9),
7		dh-systemd,
	9	debhelper (>= 9.20160709) \| dh-systemd,
8	10	dh-virtualenv (>= 1.1),
9	11	libsystemd-dev,
10	12	libpq-dev,

+11

-2

docker/Dockerfile-dhvirtualenv less more

49	49	ARG distro=""
50	50	ENV distro ${distro}
51	51
	52	# Python < 3.7 assumes LANG="C" means ASCII-only and throws on printing unicode
	53	# http://bugs.python.org/issue19846
	54	ENV LANG C.UTF-8
	55
52	56	# Install the build dependencies
53	57	#
54	58	# NB: keep this list in sync with the list of build-deps in debian/control
55	59	# TODO: it would be nice to do that automatically.
	60	# TODO: Remove the dh-systemd stanza after dropping support for Ubuntu xenial
	61	# it's a transitional package on all other, more recent releases
56	62	RUN apt-get update -qq -o Acquire::Languages=none \
57	63	&& env DEBIAN_FRONTEND=noninteractive apt-get install \
58	64	-yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
59	65	build-essential \
60	66	debhelper \
61	67	devscripts \
62		dh-systemd \
63	68	libsystemd-dev \
64	69	lsb-release \
65	70	pkg-config \

68	73	python3-setuptools \
69	74	python3-venv \
70	75	sqlite3 \
71		libpq-dev
	76	libpq-dev \
	77	xmlsec1 \
	78	&& ( env DEBIAN_FRONTEND=noninteractive apt-get install \
	79	-yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
	80	dh-systemd \|\| true )
72	81
73	82	COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /
74	83

+96

-40

docs/admin_api/media_admin_api.md less more

	0	# Contents
	1	- [List all media in a room](#list-all-media-in-a-room)
	2	- [Quarantine media](#quarantine-media)
	3	* [Quarantining media by ID](#quarantining-media-by-id)
	4	* [Quarantining media in a room](#quarantining-media-in-a-room)
	5	* [Quarantining all media of a user](#quarantining-all-media-of-a-user)
	6	- [Delete local media](#delete-local-media)
	7	* [Delete a specific local media](#delete-a-specific-local-media)
	8	* [Delete local media by date or size](#delete-local-media-by-date-or-size)
	9	- [Purge Remote Media API](#purge-remote-media-api)
	10
0	11	# List all media in a room
1	12
2	13	This API gets a list of known media in a room.

10	21	server admin: see [README.rst](README.rst).
11	22
12	23	The API returns a JSON body like the following:
13		```
14		{
15		"local": [
16		"mxc://localhost/xwvutsrqponmlkjihgfedcba",
17		"mxc://localhost/abcdefghijklmnopqrstuvwx"
18		],
19		"remote": [
20		"mxc://matrix.org/xwvutsrqponmlkjihgfedcba",
21		"mxc://matrix.org/abcdefghijklmnopqrstuvwx"
22		]
	24	```json
	25	{
	26	"local": [
	27	"mxc://localhost/xwvutsrqponmlkjihgfedcba",
	28	"mxc://localhost/abcdefghijklmnopqrstuvwx"
	29	],
	30	"remote": [
	31	"mxc://matrix.org/xwvutsrqponmlkjihgfedcba",
	32	"mxc://matrix.org/abcdefghijklmnopqrstuvwx"
	33	]
23	34	}
24	35	```
25	36

47	58
48	59	Response:
49	60
50		```
	61	```json
51	62	{}
52	63	```
53	64

67	78
68	79	Response:
69	80
70		```
71		{
72		"num_quarantined": 10 # The number of media items successfully quarantined
73		}
74		```
	81	```json
	82	{
	83	"num_quarantined": 10
	84	}
	85	```
	86
	87	The following fields are returned in the JSON response body:
	88
	89	* `num_quarantined`: integer - The number of media items successfully quarantined
75	90
76	91	Note that there is a legacy endpoint, `POST
77		/_synapse/admin/v1/quarantine_media/<room_id >`, that operates the same.
	92	/_synapse/admin/v1/quarantine_media/<room_id>`, that operates the same.
78	93	However, it is deprecated and may be removed in a future release.
79	94
80	95	## Quarantining all media of a user

91	106	{}
92	107	```
93	108
94		Where `user_id` is in the form of `@bob:example.org`.
95
96		Response:
97
98		```
99		{
100		"num_quarantined": 10 # The number of media items successfully quarantined
101		}
102		```
	109	URL Parameters
	110
	111	* `user_id`: string - User ID in the form of `@bob:example.org`
	112
	113	Response:
	114
	115	```json
	116	{
	117	"num_quarantined": 10
	118	}
	119	```
	120
	121	The following fields are returned in the JSON response body:
	122
	123	* `num_quarantined`: integer - The number of media items successfully quarantined
103	124
104	125	# Delete local media
105	126	This API deletes the local media from the disk of your own server.

107	128	remote homeservers.
108	129	This API will not affect media that has been uploaded to external
109	130	media repositories (e.g https://github.com/turt2live/matrix-media-repo/).
110		See also [purge_remote_media.rst](purge_remote_media.rst).
	131	See also [Purge Remote Media API](#purge-remote-media-api).
111	132
112	133	## Delete a specific local media
113	134	Delete a specific `media_id`.

128	149	Response:
129	150
130	151	```json
131		{
132		"deleted_media": [
133		"abcdefghijklmnopqrstuvwx"
134		],
135		"total": 1
136		}
	152	{
	153	"deleted_media": [
	154	"abcdefghijklmnopqrstuvwx"
	155	],
	156	"total": 1
	157	}
137	158	```
138	159
139	160	The following fields are returned in the JSON response body:

166	187	Response:
167	188
168	189	```json
169		{
170		"deleted_media": [
171		"abcdefghijklmnopqrstuvwx",
172		"abcdefghijklmnopqrstuvwz"
173		],
174		"total": 2
175		}
	190	{
	191	"deleted_media": [
	192	"abcdefghijklmnopqrstuvwx",
	193	"abcdefghijklmnopqrstuvwz"
	194	],
	195	"total": 2
	196	}
176	197	```
177	198
178	199	The following fields are returned in the JSON response body:
179	200
180	201	* `deleted_media`: an array of strings - List of deleted `media_id`
181	202	* `total`: integer - Total number of deleted `media_id`
	203
	204	# Purge Remote Media API
	205
	206	The purge remote media API allows server admins to purge old cached remote media.
	207
	208	The API is:
	209
	210	```
	211	POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>
	212
	213	{}
	214	```
	215
	216	URL Parameters
	217
	218	* `unix_timestamp_in_ms`: string representing a positive integer - Unix timestamp in ms.
	219	All cached media that was last accessed before this timestamp will be removed.
	220
	221	Response:
	222
	223	```json
	224	{
	225	"deleted": 10
	226	}
	227	```
	228
	229	The following fields are returned in the JSON response body:
	230
	231	* `deleted`: integer - The number of media items successfully deleted
	232
	233	To use it, you will need to authenticate by providing an `access_token` for a
	234	server admin: see [README.rst](README.rst).
	235
	236	If the user re-requests purged remote media, synapse will re-request the media
	237	from the originating server.

+0

-20

~~docs/admin_api/purge_remote_media.rst~~ less more

0		Purge Remote Media API
1		======================
2
3		The purge remote media API allows server admins to purge old cached remote
4		media.
5
6		The API is::
7
8		POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>
9
10		{}
11
12		\... which will remove all cached media that was last accessed before
13		``<unix_timestamp_in_ms>``.
14
15		To use it, you will need to authenticate by providing an ``access_token`` for a
16		server admin: see `README.rst <README.rst>`_.
17
18		If the user re-requests purged remote media, synapse will re-request the media
19		from the originating server.

+5

-4

docs/admin_api/purge_room.md less more

0		Purge room API
1		==============
	0	Deprecated: Purge room API
	1	==========================
	2
	3	**The old Purge room API is deprecated and will be removed in a future release.
	4	See the new [Delete Room API](rooms.md#delete-room-api) for more details.**
2	5
3	6	This API will remove all trace of a room from your database.
4	7
5	8	All local users must have left the room before it can be removed.
6
7		See also: [Delete Room API](rooms.md#delete-room-api)
8	9
9	10	The API is:
10	11

+69

-13

docs/admin_api/rooms.md less more

	0	# Contents
	1	- [List Room API](#list-room-api)
	2	* [Parameters](#parameters)
	3	* [Usage](#usage)
	4	- [Room Details API](#room-details-api)
	5	- [Room Members API](#room-members-api)
	6	- [Delete Room API](#delete-room-api)
	7	* [Parameters](#parameters-1)
	8	* [Response](#response)
	9	* [Undoing room shutdowns](#undoing-room-shutdowns)
	10	- [Make Room Admin API](#make-room-admin-api)
	11
0	12	# List Room API
1	13
2	14	The List Room admin API allows server admins to get a list of rooms on their

75	87
76	88	Response:
77	89
78		```
	90	```jsonc
79	91	{
80	92	"rooms": [
81	93	{

127	139
128	140	Response:
129	141
130		```
	142	```json
131	143	{
132	144	"rooms": [
133	145	{

162	174
163	175	Response:
164	176
165		```
	177	```jsonc
166	178	{
167	179	"rooms": [
168	180	{

218	230
219	231	Response:
220	232
221		```
	233	```jsonc
222	234	{
223	235	"rooms": [
224	236	{
225	237	"room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
226	238	"name": "Music Theory",
227	239	"canonical_alias": "#musictheory:matrix.org",
228		"joined_members": 127
	240	"joined_members": 127,
229	241	"joined_local_members": 2,
230	242	"version": "1",
231	243	"creator": "@foo:matrix.org",

242	254	"room_id": "!twcBhHVdZlQWuuxBhN:termina.org.uk",
243	255	"name": "weechat-matrix",
244	256	"canonical_alias": "#weechat-matrix:termina.org.uk",
245		"joined_members": 137
	257	"joined_members": 137,
246	258	"joined_local_members": 20,
247	259	"version": "4",
248	260	"creator": "@foo:termina.org.uk",

277	289	* `canonical_alias` - The canonical (main) alias address of the room.
278	290	* `joined_members` - How many users are currently in the room.
279	291	* `joined_local_members` - How many local users are currently in the room.
	292	* `joined_local_devices` - How many local devices are currently in the room.
280	293	* `version` - The version of the room as a string.
281	294	* `creator` - The `user_id` of the room creator.
282	295	* `encryption` - Algorithm of end-to-end encryption of messages. Is `null` if encryption is not active.

299	312
300	313	Response:
301	314
302		```
	315	```json
303	316	{
304	317	"room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
305	318	"name": "Music Theory",
306	319	"avatar": "mxc://matrix.org/AQDaVFlbkQoErdOgqWRgiGSV",
307	320	"topic": "Theory, Composition, Notation, Analysis",
308	321	"canonical_alias": "#musictheory:matrix.org",
309		"joined_members": 127
	322	"joined_members": 127,
310	323	"joined_local_members": 2,
	324	"joined_local_devices": 2,
311	325	"version": "1",
312	326	"creator": "@foo:matrix.org",
313	327	"encryption": null,

341	355
342	356	Response:
343	357
344		```
	358	```json
345	359	{
346	360	"members": [
347	361	"@foo:matrix.org",
348	362	"@bar:matrix.org",
349		"@foobar:matrix.org
350		],
	363	"@foobar:matrix.org"
	364	],
351	365	"total": 3
352	366	}
353	367	```

356	370
357	371	The Delete Room admin API allows server admins to remove rooms from server
358	372	and block these rooms.
359		It is a combination and improvement of "[Shutdown room](shutdown_room.md)"
360		and "[Purge room](purge_room.md)" API.
361	373
362	374	Shuts down a room. Moves all local users and room aliases automatically to a
363	375	new room if `new_room_user_id` is set. Otherwise local users only

454	466	* `local_aliases` - An array of strings representing the local aliases that were migrated from
455	467	the old room to the new.
456	468	* `new_room_id` - A string representing the room ID of the new room.
	469
	470
	471	## Undoing room shutdowns
	472
	473	Note: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
	474	the structure can and does change without notice.
	475
	476	First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
	477	never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
	478	to recover at all:
	479
	480	* If the room was invite-only, your users will need to be re-invited.
	481	* If the room no longer has any members at all, it'll be impossible to rejoin.
	482	* The first user to rejoin will have to do so via an alias on a different server.
	483
	484	With all that being said, if you still want to try and recover the room:
	485
	486	1. For safety reasons, shut down Synapse.
	487	2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
	488	* For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
	489	* The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
	490	3. Restart Synapse.
	491
	492	You will have to manually handle, if you so choose, the following:
	493
	494	* Aliases that would have been redirected to the Content Violation room.
	495	* Users that would have been booted from the room (and will have been force-joined to the Content Violation room).
	496	* Removal of the Content Violation room if desired.
	497
	498
	499	# Make Room Admin API
	500
	501	Grants another user the highest power available to a local user who is in the room.
	502	If the user is not in the room, and it is not publicly joinable, then invite the user.
	503
	504	By default the server admin (the caller) is granted power, but another user can
	505	optionally be specified, e.g.:
	506
	507	```
	508	POST /_synapse/admin/v1/rooms/<room_id_or_alias>/make_room_admin
	509	{
	510	"user_id": "@foo:example.com"
	511	}
	512	```

+4

-3

docs/admin_api/shutdown_room.md less more

0		# Shutdown room API
	0	# Deprecated: Shutdown room API
	1
	2	**The old Shutdown room API is deprecated and will be removed in a future release.
	3	See the new [Delete Room API](rooms.md#delete-room-api) for more details.**
1	4
2	5	Shuts down a room, preventing new joins and moves local users and room aliases automatically
3	6	to a new room. The new room will be created with the user specified by the

8	11
9	12	The local server will only have the power to move local user and room aliases to
10	13	the new room. Users on other servers will be unaffected.
11
12		See also: [Delete Room API](rooms.md#delete-room-api)
13	14
14	15	## API
15	16

+6

-3

docs/admin_api/user_admin_api.rst less more

29	29	],
30	30	"avatar_url": "<avatar_url>",
31	31	"admin": false,
32		"deactivated": false
	32	"deactivated": false,
	33	"password_hash": "$2b$12$p9B4GkqYdRTPGD",
	34	"creation_ts": 1560432506,
	35	"appservice_id": null,
	36	"consent_server_notice_sent": null,
	37	"consent_version": null
33	38	}
34	39
35	40	URL parameters:

138	143	"users": [
139	144	{
140	145	"name": "<user_id1>",
141		"password_hash": "<password_hash1>",
142	146	"is_guest": 0,
143	147	"admin": 0,
144	148	"user_type": null,

147	151	"avatar_url": null
148	152	}, {
149	153	"name": "<user_id2>",
150		"password_hash": "<password_hash2>",
151	154	"is_guest": 0,
152	155	"admin": 1,
153	156	"user_type": null,

+3

-3

docs/dev/cas.md less more

30	30	You should now have a Django project configured to serve CAS authentication with
31	31	a single user created.
32	32
33		## Configure Synapse (and Riot) to use CAS
	33	## Configure Synapse (and Element) to use CAS
34	34
35	35	1. Modify your `homeserver.yaml` to enable CAS and point it to your locally
36	36	running Django test server:

50	50
51	51	## Testing the configuration
52	52
53		Then in Riot:
	53	Then in Element:
54	54
55		1. Visit the login page with a Riot pointing at your homeserver.
	55	1. Visit the login page with a Element pointing at your homeserver.
56	56	2. Click the Single Sign-On button.
57	57	3. Login using the credentials created with `createsuperuser`.
58	58	4. You should be logged in.

+84

-33

docs/sample_config.yaml less more

143	143	#
144	144	#enable_search: false
145	145
	146	# Prevent outgoing requests from being sent to the following blacklisted IP address
	147	# CIDR ranges. If this option is not specified then it defaults to private IP
	148	# address ranges (see the example below).
	149	#
	150	# The blacklist applies to the outbound requests for federation, identity servers,
	151	# push servers, and for checking key validity for third-party invite events.
	152	#
	153	# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
	154	# listed here, since they correspond to unroutable addresses.)
	155	#
	156	# This option replaces federation_ip_range_blacklist in Synapse v1.25.0.
	157	#
	158	#ip_range_blacklist:
	159	# - '127.0.0.0/8'
	160	# - '10.0.0.0/8'
	161	# - '172.16.0.0/12'
	162	# - '192.168.0.0/16'
	163	# - '100.64.0.0/10'
	164	# - '192.0.0.0/24'
	165	# - '169.254.0.0/16'
	166	# - '198.18.0.0/15'
	167	# - '192.0.2.0/24'
	168	# - '198.51.100.0/24'
	169	# - '203.0.113.0/24'
	170	# - '224.0.0.0/4'
	171	# - '::1/128'
	172	# - 'fe80::/10'
	173	# - 'fc00::/7'
	174
	175	# List of IP address CIDR ranges that should be allowed for federation,
	176	# identity servers, push servers, and for checking key validity for
	177	# third-party invite events. This is useful for specifying exceptions to
	178	# wide-ranging blacklisted target IP ranges - e.g. for communication with
	179	# a push server only visible in your network.
	180	#
	181	# This whitelist overrides ip_range_blacklist and defaults to an empty
	182	# list.
	183	#
	184	#ip_range_whitelist:
	185	# - '192.168.1.1'
	186
146	187	# List of ports that Synapse should listen on, their purpose and their
147	188	# configuration.
148	189	#

640	681	# - lon.example.com
641	682	# - nyc.example.com
642	683	# - syd.example.com
643
644		# Prevent federation requests from being sent to the following
645		# blacklist IP address CIDR ranges. If this option is not specified, or
646		# specified with an empty list, no ip range blacklist will be enforced.
647		#
648		# As of Synapse v1.4.0 this option also affects any outbound requests to identity
649		# servers provided by user input.
650		#
651		# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
652		# listed here, since they correspond to unroutable addresses.)
653		#
654		federation_ip_range_blacklist:
655		- '127.0.0.0/8'
656		- '10.0.0.0/8'
657		- '172.16.0.0/12'
658		- '192.168.0.0/16'
659		- '100.64.0.0/10'
660		- '169.254.0.0/16'
661		- '::1/128'
662		- 'fe80::/64'
663		- 'fc00::/7'
664	684
665	685	# Report prometheus metrics on the age of PDUs being sent to and received from
666	686	# the following domains. This can be used to give an idea of "delay" on inbound

952	972	# - '172.16.0.0/12'
953	973	# - '192.168.0.0/16'
954	974	# - '100.64.0.0/10'
	975	# - '192.0.0.0/24'
955	976	# - '169.254.0.0/16'
	977	# - '198.18.0.0/15'
	978	# - '192.0.2.0/24'
	979	# - '198.51.100.0/24'
	980	# - '203.0.113.0/24'
	981	# - '224.0.0.0/4'
956	982	# - '::1/128'
957		# - 'fe80::/64'
	983	# - 'fe80::/10'
958	984	# - 'fc00::/7'
959	985
960	986	# List of IP address CIDR ranges that the URL preview spider is allowed

1798	1824	# * user: The claims returned by the UserInfo Endpoint and/or in the ID
1799	1825	# Token
1800	1826	#
1801		# This must be configured if using the default mapping provider.
	1827	# If this is not set, the user will be prompted to choose their
	1828	# own username.
1802	1829	#
1803		localpart_template: "{{ user.preferred_username }}"
	1830	#localpart_template: "{{ user.preferred_username }}"
1804	1831
1805	1832	# Jinja2 template for the display name to set on first login.
1806	1833	#

1876	1903	# - https://my.custom.client/
1877	1904
1878	1905	# Directory in which Synapse will try to find the template files below.
1879		# If not set, default templates from within the Synapse package will be used.
1880		#
1881		# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
1882		# If you do uncomment it, you will need to make sure that all the templates
1883		# below are in the directory.
	1906	# If not set, or the files named below are not found within the template
	1907	# directory, default templates from within the Synapse package will be used.
1884	1908	#
1885	1909	# Synapse will look for the following templates in this directory:
1886	1910	#

2044	2068	#
2045	2069	#require_uppercase: true
2046	2070
	2071	ui_auth:
	2072	# The number of milliseconds to allow a user-interactive authentication
	2073	# session to be active.
	2074	#
	2075	# This defaults to 0, meaning the user is queried for their credentials
	2076	# before every action, but this can be overridden to alow a single
	2077	# validation to be re-used. This weakens the protections afforded by
	2078	# the user-interactive authentication process, by allowing for multiple
	2079	# (and potentially different) operations to use the same validation session.
	2080	#
	2081	# Uncomment below to allow for credential validation to last for 15
	2082	# seconds.
	2083	#
	2084	#session_timeout: 15000
	2085
2047	2086
2048	2087	# Configuration for sending emails from Synapse.
2049	2088	#

2109	2148	#
2110	2149	#validation_token_lifetime: 15m
2111	2150
	2151	# The web client location to direct users to during an invite. This is passed
	2152	# to the identity server as the org.matrix.web_client_location key. Defaults
	2153	# to unset, giving no guidance to the identity server.
	2154	#
	2155	#invite_client_location: https://app.element.io
	2156
2112	2157	# Directory in which Synapse will try to find the template files below.
2113		# If not set, default templates from within the Synapse package will be used.
2114		#
2115		# Do not uncomment this setting unless you want to customise the templates.
	2158	# If not set, or the files named below are not found within the template
	2159	# directory, default templates from within the Synapse package will be used.
2116	2160	#
2117	2161	# Synapse will look for the following templates in this directory:
2118	2162	#

2321	2365	# If enabled, non server admins can only create groups with local parts
2322	2366	# starting with this prefix
2323	2367	#
2324		#group_creation_prefix: "unofficial/"
	2368	#group_creation_prefix: "unofficial_"
2325	2369
2326	2370
2327	2371

2586	2630	#
2587	2631	#run_background_tasks_on: worker1
2588	2632
	2633	# A shared secret used by the replication APIs to authenticate HTTP requests
	2634	# from workers.
	2635	#
	2636	# By default this is unused and traffic is not authenticated.
	2637	#
	2638	#worker_replication_secret: ""
	2639
2589	2640
2590	2641	# Configuration for Redis when using workers. This must be enabled when
2591	2642	# using workers (unless using old style direct TCP configuration).

+13

-6

docs/spam_checker.md less more

21	21	* `user_may_create_room`
22	22	* `user_may_create_room_alias`
23	23	* `user_may_publish_room`
	24	* `check_username_for_spam`
	25	* `check_registration_for_spam`
24	26
25	27	The details of the each of these methods (as well as their inputs and outputs)
26	28	are documented in the `synapse.events.spamcheck.SpamChecker` class.

31	33	### Example
32	34
33	35	```python
	36	from synapse.spam_checker_api import RegistrationBehaviour
	37
34	38	class ExampleSpamChecker:
35	39	def __init__(self, config, api):
36	40	self.config = config
37	41	self.api = api
38	42
39		def check_event_for_spam(self, foo):
	43	async def check_event_for_spam(self, foo):
40	44	return False # allow all events
41	45
42		def user_may_invite(self, inviter_userid, invitee_userid, room_id):
	46	async def user_may_invite(self, inviter_userid, invitee_userid, room_id):
43	47	return True # allow all invites
44	48
45		def user_may_create_room(self, userid):
	49	async def user_may_create_room(self, userid):
46	50	return True # allow all room creations
47	51
48		def user_may_create_room_alias(self, userid, room_alias):
	52	async def user_may_create_room_alias(self, userid, room_alias):
49	53	return True # allow all room aliases
50	54
51		def user_may_publish_room(self, userid, room_id):
	55	async def user_may_publish_room(self, userid, room_id):
52	56	return True # allow publishing of all rooms
53	57
54		def check_username_for_spam(self, user_profile):
	58	async def check_username_for_spam(self, user_profile):
55	59	return False # allow all usernames
	60
	61	async def check_registration_for_spam(self, email_threepid, username, request_info):
	62	return RegistrationBehaviour.ALLOW # allow all registrations
56	63	```
57	64
58	65	## Configuration

+21

-11

docs/sso_mapping_providers.md less more

14	14	SSO mapping providers are currently supported for OpenID and SAML SSO
15	15	configurations. Please see the details below for how to implement your own.
16	16
17		It is the responsibility of the mapping provider to normalise the SSO attributes
18		and map them to a valid Matrix ID. The
19		[specification for Matrix IDs](https://matrix.org/docs/spec/appendices#user-identifiers)
20		has some information about what is considered valid. Alternately an easy way to
21		ensure it is valid is to use a Synapse utility function:
22		`synapse.types.map_username_to_mxid_localpart`.
	17	It is up to the mapping provider whether the user should be assigned a predefined
	18	Matrix ID based on the SSO attributes, or if the user should be allowed to
	19	choose their own username.
	20
	21	In the first case - where users are automatically allocated a Matrix ID - it is
	22	the responsibility of the mapping provider to normalise the SSO attributes and
	23	map them to a valid Matrix ID. The [specification for Matrix
	24	IDs](https://matrix.org/docs/spec/appendices#user-identifiers) has some
	25	information about what is considered valid.
	26
	27	If the mapping provider does not assign a Matrix ID, then Synapse will
	28	automatically serve an HTML page allowing the user to pick their own username.
23	29
24	30	External mapping providers are provided to Synapse in the form of an external
25	31	Python module. You can retrieve this module from [PyPI](https://pypi.org) or elsewhere,

79	85	with failures=1. The method should then return a different
80	86	`localpart` value, such as `john.doe1`.
81	87	- Returns a dictionary with two keys:
82		- localpart: A required string, used to generate the Matrix ID.
83		- displayname: An optional string, the display name for the user.
	88	- `localpart`: A string, used to generate the Matrix ID. If this is
	89	`None`, the user is prompted to pick their own username.
	90	- `displayname`: An optional string, the display name for the user.
84	91	* `get_extra_attributes(self, userinfo, token)`
85	92	- This method must be async.
86	93	- Arguments:

115	122
116	123	A custom mapping provider must specify the following methods:
117	124
118		* `__init__(self, parsed_config)`
	125	* `__init__(self, parsed_config, module_api)`
119	126	- Arguments:
120	127	- `parsed_config` - A configuration object that is the return value of the
121	128	`parse_config` method. You should set any configuration options needed by
122	129	the module here.
	130	- `module_api` - a `synapse.module_api.ModuleApi` object which provides the
	131	stable API available for extension modules.
123	132	* `parse_config(config)`
124	133	- This method should have the `@staticmethod` decoration.
125	134	- Arguments:

162	171	redirected to.
163	172	- This method must return a dictionary, which will then be used by Synapse
164	173	to build a new user. The following keys are allowed:
165		* `mxid_localpart` - Required. The mxid localpart of the new user.
	174	* `mxid_localpart` - The mxid localpart of the new user. If this is
	175	`None`, the user is prompted to pick their own username.
166	176	* `displayname` - The displayname of the new user. If not provided, will default to
167	177	the value of `mxid_localpart`.
168	178	* `emails` - A list of emails for the new user. If not provided, will
169	179	default to an empty list.
170
	180
171	181	Alternatively it can raise a `synapse.api.errors.RedirectException` to
172	182	redirect the user to another page. This is useful to prompt the user for
173	183	additional information, e.g. if you want them to provide their own username.

+6

-1

docs/workers.md less more

88	88	Normally, only a couple of changes are needed to make an existing configuration
89	89	file suitable for use with workers. First, you need to enable an "HTTP replication
90	90	listener" for the main process; and secondly, you need to enable redis-based
91		replication. For example:
	91	replication. Optionally, a shared secret can be used to authenticate HTTP
	92	traffic between workers. For example:
92	93
93	94
94	95	```yaml

101	102	type: http
102	103	resources:
103	104	- names: [replication]
	105
	106	# Add a random shared secret to authenticate traffic.
	107	worker_replication_secret: ""
104	108
105	109	redis:
106	110	enabled: true

224	228	^/_matrix/client/(r0\|unstable)/auth/.*/fallback/web$
225	229
226	230	# Event sending requests
	231	^/_matrix/client/(api/v1\|r0\|unstable)/rooms/.*/redact
227	232	^/_matrix/client/(api/v1\|r0\|unstable)/rooms/.*/send
228	233	^/_matrix/client/(api/v1\|r0\|unstable)/rooms/.*/state/
229	234	^/_matrix/client/(api/v1\|r0\|unstable)/rooms/.*/(join\|invite\|leave\|ban\|unban\|kick)$

+27

-2

mypy.ini less more

6	6	show_traceback = True
7	7	mypy_path = stubs
8	8	warn_unreachable = True
	9
	10	# To find all folders that pass mypy you run:
	11	#
	12	# find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null" \; -print
	13
9	14	files =
10	15	scripts-dev/sign_json,
11	16	synapse/api,
12	17	synapse/appservice,
13	18	synapse/config,
	19	synapse/crypto,
14	20	synapse/event_auth.py,
15	21	synapse/events/builder.py,
16	22	synapse/events/validator.py,

19	25	synapse/handlers/_base.py,
20	26	synapse/handlers/account_data.py,
21	27	synapse/handlers/account_validity.py,
	28	synapse/handlers/admin.py,
22	29	synapse/handlers/appservice.py,
23	30	synapse/handlers/auth.py,
24	31	synapse/handlers/cas_handler.py,

37	44	synapse/handlers/presence.py,
38	45	synapse/handlers/profile.py,
39	46	synapse/handlers/read_marker.py,
	47	synapse/handlers/receipts.py,
40	48	synapse/handlers/register.py,
41	49	synapse/handlers/room.py,
	50	synapse/handlers/room_list.py,
42	51	synapse/handlers/room_member.py,
43	52	synapse/handlers/room_member_worker.py,
44	53	synapse/handlers/saml_handler.py,
	54	synapse/handlers/sso.py,
45	55	synapse/handlers/sync.py,
	56	synapse/handlers/user_directory.py,
46	57	synapse/handlers/ui_auth,
47	58	synapse/http/client.py,
48	59	synapse/http/federation/matrix_federation_agent.py,

54	65	synapse/metrics,
55	66	synapse/module_api,
56	67	synapse/notifier.py,
57		synapse/push/pusherpool.py,
58		synapse/push/push_rule_evaluator.py,
	68	synapse/push,
59	69	synapse/replication,
60	70	synapse/rest,
61	71	synapse/server.py,
62	72	synapse/server_notices,
63	73	synapse/spam_checker_api,
64	74	synapse/state,
	75	synapse/storage/__init__.py,
	76	synapse/storage/_base.py,
	77	synapse/storage/background_updates.py,
65	78	synapse/storage/databases/main/appservice.py,
66	79	synapse/storage/databases/main/events.py,
	80	synapse/storage/databases/main/keys.py,
	81	synapse/storage/databases/main/pusher.py,
67	82	synapse/storage/databases/main/registration.py,
68	83	synapse/storage/databases/main/stream.py,
69	84	synapse/storage/databases/main/ui_auth.py,
70	85	synapse/storage/database.py,
71	86	synapse/storage/engines,
	87	synapse/storage/keys.py,
72	88	synapse/storage/persist_events.py,
	89	synapse/storage/prepare_database.py,
	90	synapse/storage/purge_events.py,
	91	synapse/storage/push_rule.py,
	92	synapse/storage/relations.py,
	93	synapse/storage/roommember.py,
73	94	synapse/storage/state.py,
	95	synapse/storage/types.py,
74	96	synapse/storage/util,
75	97	synapse/streams,
76	98	synapse/types.py,

105	127	ignore_missing_imports = True
106	128
107	129	[mypy-h11]
	130	ignore_missing_imports = True
	131
	132	[mypy-msgpack]
108	133	ignore_missing_imports = True
109	134
110	135	[mypy-opentracing]

+3

-1

scripts/generate_log_config less more

39	39	)
40	40
41	41	args = parser.parse_args()
42		args.output_file.write(DEFAULT_LOG_CONFIG.substitute(log_file=args.log_file))
	42	out = args.output_file
	43	out.write(DEFAULT_LOG_CONFIG.substitute(log_file=args.log_file))
	44	out.flush()

+2

-0

scripts-dev/mypy_synapse_plugin.py less more

30	30	) -> Optional[Callable[[MethodSigContext], CallableType]]:
31	31	if fullname.startswith(
32	32	"synapse.util.caches.descriptors._CachedFunction.__call__"
	33	) or fullname.startswith(
	34	"synapse.util.caches.descriptors._LruCachedFunction.__call__"
33	35	):
34	36	return cached_function_method_signature
35	37	return None

+1

-1

synapse/__init__.py less more

47	47	except ImportError:
48	48	pass
49	49
50		__version__ = "1.24.0"
	50	__version__ = "1.25.0"
51	51
52	52	if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
53	53	# We import here so that we don't have to install a bunch of deps when

+6

-3

synapse/api/auth.py less more

22	22	import synapse.types
23	23	from synapse import event_auth
24	24	from synapse.api.auth_blocking import AuthBlocking
25		from synapse.api.constants import EventTypes, Membership
	25	from synapse.api.constants import EventTypes, HistoryVisibility, Membership
26	26	from synapse.api.errors import (
27	27	AuthError,
28	28	Codes,

30	30	MissingClientTokenError,
31	31	)
32	32	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
	33	from synapse.appservice import ApplicationService
33	34	from synapse.events import EventBase
	35	from synapse.http.site import SynapseRequest
34	36	from synapse.logging import opentracing as opentracing
35	37	from synapse.storage.databases.main.registration import TokenLookupResult
36	38	from synapse.types import StateMap, UserID

473	475	now = self.hs.get_clock().time_msec()
474	476	return now < expiry
475	477
476		def get_appservice_by_req(self, request):
	478	def get_appservice_by_req(self, request: SynapseRequest) -> ApplicationService:
477	479	token = self.get_access_token_from_request(request)
478	480	service = self.store.get_app_service_by_token(token)
479	481	if not service:

645	647	)
646	648	if (
647	649	visibility
648		and visibility.content["history_visibility"] == "world_readable"
	650	and visibility.content.get("history_visibility")
	651	== HistoryVisibility.WORLD_READABLE
649	652	):
650	653	return Membership.JOIN, None
651	654	raise AuthError(

+7

-0

synapse/api/auth_blocking.py less more

35	35	self._limit_usage_by_mau = hs.config.limit_usage_by_mau
36	36	self._mau_limits_reserved_threepids = hs.config.mau_limits_reserved_threepids
37	37	self._server_name = hs.hostname
	38	self._track_appservice_user_ips = hs.config.appservice.track_appservice_user_ips
38	39
39	40	async def check_auth_blocking(
40	41	self,

74	75	elif requester.authenticated_entity == self._server_name:
75	76	# We never block the server from doing actions on behalf of
76	77	# users.
	78	return
	79	elif requester.app_service and not self._track_appservice_user_ips:
	80	# If we're authenticated as an appservice then we only block
	81	# auth if `track_appservice_user_ips` is set, as that option
	82	# implicitly means that application services are part of MAU
	83	# limits.
77	84	return
78	85
79	86	# Never fail an auth check for the server notices users or support user

+9

-0

synapse/api/constants.py less more

94	94
95	95	Presence = "m.presence"
96	96
	97	Dummy = "org.matrix.dummy_event"
	98
97	99
98	100	class RejectedReason:
99	101	AUTH_ERROR = "auth_error"

159	161	class AccountDataTypes:
160	162	DIRECT = "m.direct"
161	163	IGNORED_USER_LIST = "m.ignored_user_list"
	164
	165
	166	class HistoryVisibility:
	167	INVITED = "invited"
	168	JOINED = "joined"
	169	SHARED = "shared"
	170	WORLD_READABLE = "world_readable"

+5

-1

synapse/app/_base.py less more

244	244	# Set up the SIGHUP machinery.
245	245	if hasattr(signal, "SIGHUP"):
246	246
	247	reactor = hs.get_reactor()
	248
247	249	@wrap_as_background_process("sighup")
248	250	def handle_sighup(args, *kwargs):
249	251	# Tell systemd our state, if we're using it. This will silently fail if

259	261	# is so that we're in a sane state, e.g. flushing the logs may fail
260	262	# if the sighup happens in the middle of writing a log entry.
261	263	def run_sighup(args, *kwargs):
262		hs.get_clock().call_later(0, handle_sighup, args, *kwargs)
	264	# `callFromThread` should be "signal safe" as well as thread
	265	# safe.
	266	reactor.callFromThread(handle_sighup, args, *kwargs)
263	267
264	268	signal.signal(signal.SIGHUP, run_sighup)
265	269

+4

-28

synapse/app/generic_worker.py less more

88	88	ToDeviceStream,
89	89	)
90	90	from synapse.rest.admin import register_servlets_for_media_repo
91		from synapse.rest.client.v1 import events
	91	from synapse.rest.client.v1 import events, room
92	92	from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
93	93	from synapse.rest.client.v1.login import LoginRestServlet
94	94	from synapse.rest.client.v1.profile import (

97	97	ProfileRestServlet,
98	98	)
99	99	from synapse.rest.client.v1.push_rule import PushRuleRestServlet
100		from synapse.rest.client.v1.room import (
101		JoinedRoomMemberListRestServlet,
102		JoinRoomAliasServlet,
103		PublicRoomListRestServlet,
104		RoomEventContextServlet,
105		RoomInitialSyncRestServlet,
106		RoomMemberListRestServlet,
107		RoomMembershipRestServlet,
108		RoomMessageListRestServlet,
109		RoomSendEventRestServlet,
110		RoomStateEventRestServlet,
111		RoomStateRestServlet,
112		RoomTypingRestServlet,
113		)
114	100	from synapse.rest.client.v1.voip import VoipRestServlet
115	101	from synapse.rest.client.v2_alpha import groups, sync, user_directory
116	102	from synapse.rest.client.v2_alpha._base import client_patterns

265	251	super().__init__(hs)
266	252	self.hs = hs
267	253	self.is_mine_id = hs.is_mine_id
268		self.http_client = hs.get_simple_http_client()
269	254
270	255	self._presence_enabled = hs.config.use_presence
271	256

512	497	elif name == "client":
513	498	resource = JsonResource(self, canonical_json=False)
514	499
515		PublicRoomListRestServlet(self).register(resource)
516		RoomMemberListRestServlet(self).register(resource)
517		JoinedRoomMemberListRestServlet(self).register(resource)
518		RoomStateRestServlet(self).register(resource)
519		RoomEventContextServlet(self).register(resource)
520		RoomMessageListRestServlet(self).register(resource)
521	500	RegisterRestServlet(self).register(resource)
522	501	LoginRestServlet(self).register(resource)
523	502	ThreepidRestServlet(self).register(resource)

526	505	VoipRestServlet(self).register(resource)
527	506	PushRuleRestServlet(self).register(resource)
528	507	VersionsRestServlet(self).register(resource)
529		RoomSendEventRestServlet(self).register(resource)
530		RoomMembershipRestServlet(self).register(resource)
531		RoomStateEventRestServlet(self).register(resource)
532		JoinRoomAliasServlet(self).register(resource)
	508
533	509	ProfileAvatarURLRestServlet(self).register(resource)
534	510	ProfileDisplaynameRestServlet(self).register(resource)
535	511	ProfileRestServlet(self).register(resource)
536	512	KeyUploadServlet(self).register(resource)
537	513	AccountDataServlet(self).register(resource)
538	514	RoomAccountDataServlet(self).register(resource)
539		RoomTypingRestServlet(self).register(resource)
540	515
541	516	sync.register_servlets(self, resource)
542	517	events.register_servlets(self, resource)
	518	room.register_servlets(self, resource, True)
	519	room.register_deprecated_servlets(self, resource)
543	520	InitialSyncRestServlet(self).register(resource)
544		RoomInitialSyncRestServlet(self).register(resource)
545	521
546	522	user_directory.register_servlets(self, resource)
547	523

+44

-4

synapse/app/homeserver.py less more

18	18	import logging
19	19	import os
20	20	import sys
21		from typing import Iterable
	21	from typing import Iterable, Iterator
22	22
23	23	from twisted.application import service
24	24	from twisted.internet import defer, reactor

62	62	from synapse.rest.admin import AdminRestResource
63	63	from synapse.rest.health import HealthResource
64	64	from synapse.rest.key.v2 import KeyApiV2Resource
	65	from synapse.rest.synapse.client.pick_username import pick_username_resource
65	66	from synapse.rest.well_known import WellKnownResource
66	67	from synapse.server import HomeServer
67	68	from synapse.storage import DataStore

89	90	tls = listener_config.tls
90	91	site_tag = listener_config.http_options.tag
91	92	if site_tag is None:
92		site_tag = port
	93	site_tag = str(port)
93	94
94	95	# We always include a health resource.
95	96	resources = {"/health": HealthResource()}

106	107	logger.debug("Configuring additional resources: %r", additional_resources)
107	108	module_api = self.get_module_api()
108	109	for path, resmodule in additional_resources.items():
109		handler_cls, config = load_module(resmodule)
	110	handler_cls, config = load_module(
	111	resmodule,
	112	("listeners", site_tag, "additional_resources", "<%s>" % (path,)),
	113	)
110	114	handler = handler_cls(config, module_api)
111	115	if IResource.providedBy(handler):
112	116	resource = handler

188	192	"/_matrix/client/versions": client_resource,
189	193	"/.well-known/matrix/client": WellKnownResource(self),
190	194	"/_synapse/admin": AdminRestResource(self),
	195	"/_synapse/client/pick_username": pick_username_resource(self),
191	196	}
192	197	)
193	198

341	346	"Synapse Homeserver", config_options
342	347	)
343	348	except ConfigError as e:
344		sys.stderr.write("\nERROR: %s\n" % (e,))
	349	sys.stderr.write("\n")
	350	for f in format_config_error(e):
	351	sys.stderr.write(f)
	352	sys.stderr.write("\n")
345	353	sys.exit(1)
346	354
347	355	if not config:

444	452	return hs
445	453
446	454
	455	def format_config_error(e: ConfigError) -> Iterator[str]:
	456	"""
	457	Formats a config error neatly
	458
	459	The idea is to format the immediate error, plus the "causes" of those errors,
	460	hopefully in a way that makes sense to the user. For example:
	461
	462	Error in configuration at 'oidc_config.user_mapping_provider.config.display_name_template':
	463	Failed to parse config for module 'JinjaOidcMappingProvider':
	464	invalid jinja template:
	465	unexpected end of template, expected 'end of print statement'.
	466
	467	Args:
	468	e: the error to be formatted
	469
	470	Returns: An iterator which yields string fragments to be formatted
	471	"""
	472	yield "Error in configuration"
	473
	474	if e.path:
	475	yield " at '%s'" % (".".join(e.path),)
	476
	477	yield ":\n %s" % (e.msg,)
	478
	479	e = e.__cause__
	480	indent = 1
	481	while e:
	482	indent += 1
	483	yield ":\n%s%s" % (" " * indent, str(e))
	484	e = e.__cause__
	485
	486
447	487	class SynapseService(service.Service):
448	488	"""
449	489	A twisted Service class that will start synapse. Used to run synapse

+12

-2

synapse/config/_base.py less more

22	22	from collections import OrderedDict
23	23	from hashlib import sha256
24	24	from textwrap import dedent
25		from typing import Any, Callable, List, MutableMapping, Optional
	25	from typing import Any, Callable, Iterable, List, MutableMapping, Optional
26	26
27	27	import attr
28	28	import jinja2

31	31
32	32
33	33	class ConfigError(Exception):
34		pass
	34	"""Represents a problem parsing the configuration
	35
	36	Args:
	37	msg: A textual description of the error.
	38	path: Where appropriate, an indication of where in the configuration
	39	the problem lies.
	40	"""
	41
	42	def __init__(self, msg: str, path: Optional[Iterable[str]] = None):
	43	self.msg = msg
	44	self.path = path
35	45
36	46
37	47	# We split these messages out to allow packages to override with package

+7

-4

synapse/config/_base.pyi less more

0		from typing import Any, List, Optional
	0	from typing import Any, Iterable, List, Optional
1	1
2	2	from synapse.config import (
3	3	api,
4	4	appservice,
	5	auth,
5	6	captcha,
6	7	cas,
7	8	consent_config,

13	14	logger,
14	15	metrics,
15	16	oidc_config,
16		password,
17	17	password_auth_providers,
18	18	push,
19	19	ratelimiting,

34	34	workers,
35	35	)
36	36
37		class ConfigError(Exception): ...
	37	class ConfigError(Exception):
	38	def __init__(self, msg: str, path: Optional[Iterable[str]] = None):
	39	self.msg = msg
	40	self.path = path
38	41
39	42	MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS: str
40	43	MISSING_REPORT_STATS_SPIEL: str

61	64	sso: sso.SSOConfig
62	65	oidc: oidc_config.OIDCConfig
63	66	jwt: jwt_config.JWTConfig
64		password: password.PasswordConfig
	67	auth: auth.AuthConfig
65	68	email: emailconfig.EmailConfig
66	69	worker: workers.WorkerConfig
67	70	authproviders: password_auth_providers.PasswordAuthProviderConfig

+23

-10

synapse/config/_util.py less more

37	37	try:
38	38	jsonschema.validate(config, json_schema)
39	39	except jsonschema.ValidationError as e:
40		# copy `config_path` before modifying it.
41		path = list(config_path)
42		for p in list(e.path):
43		if isinstance(p, int):
44		path.append("<item %i>" % p)
45		else:
46		path.append(str(p))
	40	raise json_error_to_config_error(e, config_path)
47	41
48		raise ConfigError(
49		"Unable to parse configuration: %s at %s" % (e.message, ".".join(path))
50		)
	42
	43	def json_error_to_config_error(
	44	e: jsonschema.ValidationError, config_path: Iterable[str]
	45	) -> ConfigError:
	46	"""Converts a json validation error to a user-readable ConfigError
	47
	48	Args:
	49	e: the exception to be converted
	50	config_path: the path within the config file. This will be used as a basis
	51	for the error message.
	52
	53	Returns:
	54	a ConfigError
	55	"""
	56	# copy `config_path` before modifying it.
	57	path = list(config_path)
	58	for p in list(e.path):
	59	if isinstance(p, int):
	60	path.append("<item %i>" % p)
	61	else:
	62	path.append(str(p))
	63	return ConfigError(e.message, path)

+110

-0

synapse/config/auth.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2015, 2016 OpenMarket Ltd
	2	# Copyright 2020 The Matrix.org Foundation C.I.C.
	3	#
	4	# Licensed under the Apache License, Version 2.0 (the "License");
	5	# you may not use this file except in compliance with the License.
	6	# You may obtain a copy of the License at
	7	#
	8	# http://www.apache.org/licenses/LICENSE-2.0
	9	#
	10	# Unless required by applicable law or agreed to in writing, software
	11	# distributed under the License is distributed on an "AS IS" BASIS,
	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	13	# See the License for the specific language governing permissions and
	14	# limitations under the License.
	15
	16	from ._base import Config
	17
	18
	19	class AuthConfig(Config):
	20	"""Password and login configuration
	21	"""
	22
	23	section = "auth"
	24
	25	def read_config(self, config, **kwargs):
	26	password_config = config.get("password_config", {})
	27	if password_config is None:
	28	password_config = {}
	29
	30	self.password_enabled = password_config.get("enabled", True)
	31	self.password_localdb_enabled = password_config.get("localdb_enabled", True)
	32	self.password_pepper = password_config.get("pepper", "")
	33
	34	# Password policy
	35	self.password_policy = password_config.get("policy") or {}
	36	self.password_policy_enabled = self.password_policy.get("enabled", False)
	37
	38	# User-interactive authentication
	39	ui_auth = config.get("ui_auth") or {}
	40	self.ui_auth_session_timeout = ui_auth.get("session_timeout", 0)
	41
	42	def generate_config_section(self, config_dir_path, server_name, **kwargs):
	43	return """\
	44	password_config:
	45	# Uncomment to disable password login
	46	#
	47	#enabled: false
	48
	49	# Uncomment to disable authentication against the local password
	50	# database. This is ignored if `enabled` is false, and is only useful
	51	# if you have other password_providers.
	52	#
	53	#localdb_enabled: false
	54
	55	# Uncomment and change to a secret random string for extra security.
	56	# DO NOT CHANGE THIS AFTER INITIAL SETUP!
	57	#
	58	#pepper: "EVEN_MORE_SECRET"
	59
	60	# Define and enforce a password policy. Each parameter is optional.
	61	# This is an implementation of MSC2000.
	62	#
	63	policy:
	64	# Whether to enforce the password policy.
	65	# Defaults to 'false'.
	66	#
	67	#enabled: true
	68
	69	# Minimum accepted length for a password.
	70	# Defaults to 0.
	71	#
	72	#minimum_length: 15
	73
	74	# Whether a password must contain at least one digit.
	75	# Defaults to 'false'.
	76	#
	77	#require_digit: true
	78
	79	# Whether a password must contain at least one symbol.
	80	# A symbol is any character that's not a number or a letter.
	81	# Defaults to 'false'.
	82	#
	83	#require_symbol: true
	84
	85	# Whether a password must contain at least one lowercase letter.
	86	# Defaults to 'false'.
	87	#
	88	#require_lowercase: true
	89
	90	# Whether a password must contain at least one lowercase letter.
	91	# Defaults to 'false'.
	92	#
	93	#require_uppercase: true
	94
	95	ui_auth:
	96	# The number of milliseconds to allow a user-interactive authentication
	97	# session to be active.
	98	#
	99	# This defaults to 0, meaning the user is queried for their credentials
	100	# before every action, but this can be overridden to alow a single
	101	# validation to be re-used. This weakens the protections afforded by
	102	# the user-interactive authentication process, by allowing for multiple
	103	# (and potentially different) operations to use the same validation session.
	104	#
	105	# Uncomment below to allow for credential validation to last for 15
	106	# seconds.
	107	#
	108	#session_timeout: 15000
	109	"""

+24

-3

synapse/config/emailconfig.py less more

321	321
322	322	self.email_subjects = EmailSubjectConfig(**subjects)
323	323
	324	# The invite client location should be a HTTP(S) URL or None.
	325	self.invite_client_location = email_config.get("invite_client_location") or None
	326	if self.invite_client_location:
	327	if not isinstance(self.invite_client_location, str):
	328	raise ConfigError(
	329	"Config option email.invite_client_location must be type str"
	330	)
	331	if not (
	332	self.invite_client_location.startswith("http://")
	333	or self.invite_client_location.startswith("https://")
	334	):
	335	raise ConfigError(
	336	"Config option email.invite_client_location must be a http or https URL",
	337	path=("email", "invite_client_location"),
	338	)
	339
324	340	def generate_config_section(self, config_dir_path, server_name, **kwargs):
325	341	return (
326	342	"""\

388	404	#
389	405	#validation_token_lifetime: 15m
390	406
	407	# The web client location to direct users to during an invite. This is passed
	408	# to the identity server as the org.matrix.web_client_location key. Defaults
	409	# to unset, giving no guidance to the identity server.
	410	#
	411	#invite_client_location: https://app.element.io
	412
391	413	# Directory in which Synapse will try to find the template files below.
392		# If not set, default templates from within the Synapse package will be used.
393		#
394		# Do not uncomment this setting unless you want to customise the templates.
	414	# If not set, or the files named below are not found within the template
	415	# directory, default templates from within the Synapse package will be used.
395	416	#
396	417	# Synapse will look for the following templates in this directory:
397	418	#

+1

-42

synapse/config/federation.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	from typing import Optional
16	15
17		from netaddr import IPSet
18
19		from synapse.config._base import Config, ConfigError
	16	from synapse.config._base import Config
20	17	from synapse.config._util import validate_config
21	18
22	19

34	31
35	32	for domain in federation_domain_whitelist:
36	33	self.federation_domain_whitelist[domain] = True
37
38		self.federation_ip_range_blacklist = config.get(
39		"federation_ip_range_blacklist", []
40		)
41
42		# Attempt to create an IPSet from the given ranges
43		try:
44		self.federation_ip_range_blacklist = IPSet(
45		self.federation_ip_range_blacklist
46		)
47
48		# Always blacklist 0.0.0.0, ::
49		self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
50		except Exception as e:
51		raise ConfigError(
52		"Invalid range(s) provided in federation_ip_range_blacklist: %s" % e
53		)
54	34
55	35	federation_metrics_domains = config.get("federation_metrics_domains") or []
56	36	validate_config(

75	55	# - nyc.example.com
76	56	# - syd.example.com
77	57
78		# Prevent federation requests from being sent to the following
79		# blacklist IP address CIDR ranges. If this option is not specified, or
80		# specified with an empty list, no ip range blacklist will be enforced.
81		#
82		# As of Synapse v1.4.0 this option also affects any outbound requests to identity
83		# servers provided by user input.
84		#
85		# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
86		# listed here, since they correspond to unroutable addresses.)
87		#
88		federation_ip_range_blacklist:
89		- '127.0.0.0/8'
90		- '10.0.0.0/8'
91		- '172.16.0.0/12'
92		- '192.168.0.0/16'
93		- '100.64.0.0/10'
94		- '169.254.0.0/16'
95		- '::1/128'
96		- 'fe80::/64'
97		- 'fc00::/7'
98
99	58	# Report prometheus metrics on the age of PDUs being sent to and received from
100	59	# the following domains. This can be used to give an idea of "delay" on inbound
101	60	# and outbound federation, though be aware that any delay can be due to problems

+1

-1

synapse/config/groups.py less more

31	31	# If enabled, non server admins can only create groups with local parts
32	32	# starting with this prefix
33	33	#
34		#group_creation_prefix: "unofficial/"
	34	#group_creation_prefix: "unofficial_"
35	35	"""

+2

-2

synapse/config/homeserver.py less more

16	16	from ._base import RootConfig
17	17	from .api import ApiConfig
18	18	from .appservice import AppServiceConfig
	19	from .auth import AuthConfig
19	20	from .cache import CacheConfig
20	21	from .captcha import CaptchaConfig
21	22	from .cas import CasConfig

29	30	from .logger import LoggingConfig
30	31	from .metrics import MetricsConfig
31	32	from .oidc_config import OIDCConfig
32		from .password import PasswordConfig
33	33	from .password_auth_providers import PasswordAuthProviderConfig
34	34	from .push import PushConfig
35	35	from .ratelimiting import RatelimitConfig

75	75	CasConfig,
76	76	SSOConfig,
77	77	JWTConfig,
78		PasswordConfig,
	78	AuthConfig,
79	79	EmailConfig,
80	80	PasswordAuthProviderConfig,
81	81	PushConfig,

+1

-1

synapse/config/logger.py less more

205	205	# filter options, but care must when using e.g. MemoryHandler to buffer
206	206	# writes.
207	207
208		log_context_filter = LoggingContextFilter(request="")
	208	log_context_filter = LoggingContextFilter()
209	209	log_metadata_filter = MetadataFilter({"server_name": config.server_name})
210	210	old_factory = logging.getLogRecordFactory()
211	211

+5

-4

synapse/config/oidc_config.py less more

65	65	(
66	66	self.oidc_user_mapping_provider_class,
67	67	self.oidc_user_mapping_provider_config,
68		) = load_module(ump_config)
	68	) = load_module(ump_config, ("oidc_config", "user_mapping_provider"))
69	69
70	70	# Ensure loaded user mapping module has defined all necessary methods
71	71	required_methods = [

202	202	# * user: The claims returned by the UserInfo Endpoint and/or in the ID
203	203	# Token
204	204	#
205		# This must be configured if using the default mapping provider.
206		#
207		localpart_template: "{{{{ user.preferred_username }}}}"
	205	# If this is not set, the user will be prompted to choose their
	206	# own username.
	207	#
	208	#localpart_template: "{{{{ user.preferred_username }}}}"
208	209
209	210	# Jinja2 template for the display name to set on first login.
210	211	#

+0

-90

~~synapse/config/password.py~~ less more

0		# -- coding: utf-8 --
1		# Copyright 2015, 2016 OpenMarket Ltd
2		#
3		# Licensed under the Apache License, Version 2.0 (the "License");
4		# you may not use this file except in compliance with the License.
5		# You may obtain a copy of the License at
6		#
7		# http://www.apache.org/licenses/LICENSE-2.0
8		#
9		# Unless required by applicable law or agreed to in writing, software
10		# distributed under the License is distributed on an "AS IS" BASIS,
11		# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12		# See the License for the specific language governing permissions and
13		# limitations under the License.
14
15		from ._base import Config
16
17
18		class PasswordConfig(Config):
19		"""Password login configuration
20		"""
21
22		section = "password"
23
24		def read_config(self, config, **kwargs):
25		password_config = config.get("password_config", {})
26		if password_config is None:
27		password_config = {}
28
29		self.password_enabled = password_config.get("enabled", True)
30		self.password_localdb_enabled = password_config.get("localdb_enabled", True)
31		self.password_pepper = password_config.get("pepper", "")
32
33		# Password policy
34		self.password_policy = password_config.get("policy") or {}
35		self.password_policy_enabled = self.password_policy.get("enabled", False)
36
37		def generate_config_section(self, config_dir_path, server_name, **kwargs):
38		return """\
39		password_config:
40		# Uncomment to disable password login
41		#
42		#enabled: false
43
44		# Uncomment to disable authentication against the local password
45		# database. This is ignored if `enabled` is false, and is only useful
46		# if you have other password_providers.
47		#
48		#localdb_enabled: false
49
50		# Uncomment and change to a secret random string for extra security.
51		# DO NOT CHANGE THIS AFTER INITIAL SETUP!
52		#
53		#pepper: "EVEN_MORE_SECRET"
54
55		# Define and enforce a password policy. Each parameter is optional.
56		# This is an implementation of MSC2000.
57		#
58		policy:
59		# Whether to enforce the password policy.
60		# Defaults to 'false'.
61		#
62		#enabled: true
63
64		# Minimum accepted length for a password.
65		# Defaults to 0.
66		#
67		#minimum_length: 15
68
69		# Whether a password must contain at least one digit.
70		# Defaults to 'false'.
71		#
72		#require_digit: true
73
74		# Whether a password must contain at least one symbol.
75		# A symbol is any character that's not a number or a letter.
76		# Defaults to 'false'.
77		#
78		#require_symbol: true
79
80		# Whether a password must contain at least one lowercase letter.
81		# Defaults to 'false'.
82		#
83		#require_lowercase: true
84
85		# Whether a password must contain at least one lowercase letter.
86		# Defaults to 'false'.
87		#
88		#require_uppercase: true
89		"""

+3

-2

synapse/config/password_auth_providers.py less more

35	35	providers.append({"module": LDAP_PROVIDER, "config": ldap_config})
36	36
37	37	providers.extend(config.get("password_providers") or [])
38		for provider in providers:
	38	for i, provider in enumerate(providers):
39	39	mod_name = provider["module"]
40	40
41	41	# This is for backwards compat when the ldap auth provider resided

44	44	mod_name = LDAP_PROVIDER
45	45
46	46	(provider_class, provider_config) = load_module(
47		{"module": mod_name, "config": provider["config"]}
	47	{"module": mod_name, "config": provider["config"]},
	48	("password_providers", "<item %i>" % i),
48	49	)
49	50
50	51	self.password_providers.append((provider_class, provider_config))

+12

-14

synapse/config/repository.py less more

16	16	from collections import namedtuple
17	17	from typing import Dict, List
18	18
	19	from netaddr import IPSet
	20
	21	from synapse.config.server import DEFAULT_IP_RANGE_BLACKLIST
19	22	from synapse.python_dependencies import DependencyException, check_requirements
20	23	from synapse.util.module_loader import load_module
21	24

141	144	# them to be started.
142	145	self.media_storage_providers = [] # type: List[tuple]
143	146
144		for provider_config in storage_providers:
	147	for i, provider_config in enumerate(storage_providers):
145	148	# We special case the module "file_system" so as not to need to
146	149	# expose FileStorageProviderBackend
147	150	if provider_config["module"] == "file_system":

150	153	".FileStorageProviderBackend"
151	154	)
152	155
153		provider_class, parsed_config = load_module(provider_config)
	156	provider_class, parsed_config = load_module(
	157	provider_config, ("media_storage_providers", "<item %i>" % i)
	158	)
154	159
155	160	wrapper_config = MediaStorageProviderConfig(
156	161	provider_config.get("store_local", False),

181	186	"to work"
182	187	)
183	188
184		# netaddr is a dependency for url_preview
185		from netaddr import IPSet
186
187	189	self.url_preview_ip_range_blacklist = IPSet(
188	190	config["url_preview_ip_range_blacklist"]
189	191	)

211	213	)
212	214	# strip final NL
213	215	formatted_thumbnail_sizes = formatted_thumbnail_sizes[:-1]
	216
	217	ip_range_blacklist = "\n".join(
	218	" # - '%s'" % ip for ip in DEFAULT_IP_RANGE_BLACKLIST
	219	)
214	220
215	221	return (
216	222	r"""

282	288	# you uncomment the following list as a starting point.
283	289	#
284	290	#url_preview_ip_range_blacklist:
285		# - '127.0.0.0/8'
286		# - '10.0.0.0/8'
287		# - '172.16.0.0/12'
288		# - '192.168.0.0/16'
289		# - '100.64.0.0/10'
290		# - '169.254.0.0/16'
291		# - '::1/128'
292		# - 'fe80::/64'
293		# - 'fc00::/7'
	291	%(ip_range_blacklist)s
294	292
295	293	# List of IP address CIDR ranges that the URL preview spider is allowed
296	294	# to access even if they are specified in url_preview_ip_range_blacklist.

+1

-1

synapse/config/room_directory.py less more

179	179	self._alias_regex = glob_to_regex(alias)
180	180	self._room_id_regex = glob_to_regex(room_id)
181	181	except Exception as e:
182		raise ConfigError("Failed to parse glob into regex: %s", e)
	182	raise ConfigError("Failed to parse glob into regex") from e
183	183
184	184	def matches(self, user_id, room_id, aliases):
185	185	"""Tests if this rule matches the given user_id, room_id and aliases.

+1

-1

synapse/config/saml2_config.py less more

124	124	(
125	125	self.saml2_user_mapping_provider_class,
126	126	self.saml2_user_mapping_provider_config,
127		) = load_module(ump_dict)
	127	) = load_module(ump_dict, ("saml2_config", "user_mapping_provider"))
128	128
129	129	# Ensure loaded user mapping module has defined all necessary methods
130	130	# Note parse_config() is already checked during the call to load_module

+92

-0

synapse/config/server.py less more

22	22
23	23	import attr
24	24	import yaml
	25	from netaddr import IPSet
25	26
26	27	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
27	28	from synapse.http.endpoint import parse_and_validate_server_name

37	38	# We later check for errors when binding to 0.0.0.0 and ignore them if :: is also in
38	39	# in the list.
39	40	DEFAULT_BIND_ADDRESSES = ["::", "0.0.0.0"]
	41
	42	DEFAULT_IP_RANGE_BLACKLIST = [
	43	# Localhost
	44	"127.0.0.0/8",
	45	# Private networks.
	46	"10.0.0.0/8",
	47	"172.16.0.0/12",
	48	"192.168.0.0/16",
	49	# Carrier grade NAT.
	50	"100.64.0.0/10",
	51	# Address registry.
	52	"192.0.0.0/24",
	53	# Link-local networks.
	54	"169.254.0.0/16",
	55	# Testing networks.
	56	"198.18.0.0/15",
	57	"192.0.2.0/24",
	58	"198.51.100.0/24",
	59	"203.0.113.0/24",
	60	# Multicast.
	61	"224.0.0.0/4",
	62	# Localhost
	63	"::1/128",
	64	# Link-local addresses.
	65	"fe80::/10",
	66	# Unique local addresses.
	67	"fc00::/7",
	68	]
40	69
41	70	DEFAULT_ROOM_VERSION = "6"
42	71

254	283	# Admin uri to direct users at should their instance become blocked
255	284	# due to resource constraints
256	285	self.admin_contact = config.get("admin_contact", None)
	286
	287	ip_range_blacklist = config.get(
	288	"ip_range_blacklist", DEFAULT_IP_RANGE_BLACKLIST
	289	)
	290
	291	# Attempt to create an IPSet from the given ranges
	292	try:
	293	self.ip_range_blacklist = IPSet(ip_range_blacklist)
	294	except Exception as e:
	295	raise ConfigError("Invalid range(s) provided in ip_range_blacklist.") from e
	296	# Always blacklist 0.0.0.0, ::
	297	self.ip_range_blacklist.update(["0.0.0.0", "::"])
	298
	299	try:
	300	self.ip_range_whitelist = IPSet(config.get("ip_range_whitelist", ()))
	301	except Exception as e:
	302	raise ConfigError("Invalid range(s) provided in ip_range_whitelist.") from e
	303
	304	# The federation_ip_range_blacklist is used for backwards-compatibility
	305	# and only applies to federation and identity servers. If it is not given,
	306	# default to ip_range_blacklist.
	307	federation_ip_range_blacklist = config.get(
	308	"federation_ip_range_blacklist", ip_range_blacklist
	309	)
	310	try:
	311	self.federation_ip_range_blacklist = IPSet(federation_ip_range_blacklist)
	312	except Exception as e:
	313	raise ConfigError(
	314	"Invalid range(s) provided in federation_ip_range_blacklist."
	315	) from e
	316	# Always blacklist 0.0.0.0, ::
	317	self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
257	318
258	319	if self.public_baseurl is not None:
259	320	if self.public_baseurl[-1] != "/":

560	621	def generate_config_section(
561	622	self, server_name, data_dir_path, open_private_ports, listeners, **kwargs
562	623	):
	624	ip_range_blacklist = "\n".join(
	625	" # - '%s'" % ip for ip in DEFAULT_IP_RANGE_BLACKLIST
	626	)
	627
563	628	_, bind_port = parse_and_validate_server_name(server_name)
564	629	if bind_port is not None:
565	630	unsecure_port = bind_port - 400

751	816	#
752	817	#enable_search: false
753	818
	819	# Prevent outgoing requests from being sent to the following blacklisted IP address
	820	# CIDR ranges. If this option is not specified then it defaults to private IP
	821	# address ranges (see the example below).
	822	#
	823	# The blacklist applies to the outbound requests for federation, identity servers,
	824	# push servers, and for checking key validity for third-party invite events.
	825	#
	826	# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
	827	# listed here, since they correspond to unroutable addresses.)
	828	#
	829	# This option replaces federation_ip_range_blacklist in Synapse v1.25.0.
	830	#
	831	#ip_range_blacklist:
	832	%(ip_range_blacklist)s
	833
	834	# List of IP address CIDR ranges that should be allowed for federation,
	835	# identity servers, push servers, and for checking key validity for
	836	# third-party invite events. This is useful for specifying exceptions to
	837	# wide-ranging blacklisted target IP ranges - e.g. for communication with
	838	# a push server only visible in your network.
	839	#
	840	# This whitelist overrides ip_range_blacklist and defaults to an empty
	841	# list.
	842	#
	843	#ip_range_whitelist:
	844	# - '192.168.1.1'
	845
754	846	# List of ports that Synapse should listen on, their purpose and their
755	847	# configuration.
756	848	#

+5

-4

synapse/config/spam_checker.py less more

32	32	# spam checker, and thus was simply a dictionary with module
33	33	# and config keys. Support this old behaviour by checking
34	34	# to see if the option resolves to a dictionary
35		self.spam_checkers.append(load_module(spam_checkers))
	35	self.spam_checkers.append(load_module(spam_checkers, ("spam_checker",)))
36	36	elif isinstance(spam_checkers, list):
37		for spam_checker in spam_checkers:
	37	for i, spam_checker in enumerate(spam_checkers):
	38	config_path = ("spam_checker", "<item %i>" % i)
38	39	if not isinstance(spam_checker, dict):
39		raise ConfigError("spam_checker syntax is incorrect")
	40	raise ConfigError("expected a mapping", config_path)
40	41
41		self.spam_checkers.append(load_module(spam_checker))
	42	self.spam_checkers.append(load_module(spam_checker, config_path))
42	43	else:
43	44	raise ConfigError("spam_checker syntax is incorrect")
44	45

+2

-5

synapse/config/sso.py less more

92	92	# - https://my.custom.client/
93	93
94	94	# Directory in which Synapse will try to find the template files below.
95		# If not set, default templates from within the Synapse package will be used.
96		#
97		# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
98		# If you do uncomment it, you will need to make sure that all the templates
99		# below are in the directory.
	95	# If not set, or the files named below are not found within the template
	96	# directory, default templates from within the Synapse package will be used.
100	97	#
101	98	# Synapse will look for the following templates in this directory:
102	99	#

+3

-1

synapse/config/third_party_event_rules.py less more

25	25
26	26	provider = config.get("third_party_event_rules", None)
27	27	if provider is not None:
28		self.third_party_event_rules = load_module(provider)
	28	self.third_party_event_rules = load_module(
	29	provider, ("third_party_event_rules",)
	30	)
29	31
30	32	def generate_config_section(self, **kwargs):
31	33	return """\

+10

-0

synapse/config/workers.py less more

84	84	# The port on the main synapse for HTTP replication endpoint
85	85	self.worker_replication_http_port = config.get("worker_replication_http_port")
86	86
	87	# The shared secret used for authentication when connecting to the main synapse.
	88	self.worker_replication_secret = config.get("worker_replication_secret", None)
	89
87	90	self.worker_name = config.get("worker_name", self.worker_app)
88	91
89	92	self.worker_main_http_uri = config.get("worker_main_http_uri", None)

184	187	# data). If not provided this defaults to the main process.
185	188	#
186	189	#run_background_tasks_on: worker1
	190
	191	# A shared secret used by the replication APIs to authenticate HTTP requests
	192	# from workers.
	193	#
	194	# By default this is unused and traffic is not authenticated.
	195	#
	196	#worker_replication_secret: ""
187	197	"""
188	198
189	199	def read_arguments(self, args):

+1

-1

synapse/crypto/context_factory.py less more

226	226
227	227	# This code is based on twisted.internet.ssl.ClientTLSOptions.
228	228
229		def __init__(self, hostname: bytes, verify_certs):
	229	def __init__(self, hostname: bytes, verify_certs: bool):
230	230	self._verify_certs = verify_certs
231	231
232	232	_decoded = hostname.decode("ascii")

+18

-11

synapse/crypto/event_signing.py less more

17	17	import collections.abc
18	18	import hashlib
19	19	import logging
20		from typing import Dict
	20	from typing import Any, Callable, Dict, Tuple
21	21
22	22	from canonicaljson import encode_canonical_json
23	23	from signedjson.sign import sign_json

26	26
27	27	from synapse.api.errors import Codes, SynapseError
28	28	from synapse.api.room_versions import RoomVersion
	29	from synapse.events import EventBase
29	30	from synapse.events.utils import prune_event, prune_event_dict
30	31	from synapse.types import JsonDict
31	32
32	33	logger = logging.getLogger(__name__)
33	34
	35	Hasher = Callable[[bytes], "hashlib._Hash"]
34	36
35		def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
	37
	38	def check_event_content_hash(
	39	event: EventBase, hash_algorithm: Hasher = hashlib.sha256
	40	) -> bool:
36	41	"""Check whether the hash for this PDU matches the contents"""
37	42	name, expected_hash = compute_content_hash(event.get_pdu_json(), hash_algorithm)
38	43	logger.debug(

66	71	return message_hash_bytes == expected_hash
67	72
68	73
69		def compute_content_hash(event_dict, hash_algorithm):
	74	def compute_content_hash(
	75	event_dict: Dict[str, Any], hash_algorithm: Hasher
	76	) -> Tuple[str, bytes]:
70	77	"""Compute the content hash of an event, which is the hash of the
71	78	unredacted event.
72	79
73	80	Args:
74		event_dict (dict): The unredacted event as a dict
	81	event_dict: The unredacted event as a dict
75	82	hash_algorithm: A hasher from `hashlib`, e.g. hashlib.sha256, to use
76	83	to hash the event
77	84
78	85	Returns:
79		tuple[str, bytes]: A tuple of the name of hash and the hash as raw
80		bytes.
	86	A tuple of the name of hash and the hash as raw bytes.
81	87	"""
82	88	event_dict = dict(event_dict)
83	89	event_dict.pop("age_ts", None)

93	99	return hashed.name, hashed.digest()
94	100
95	101
96		def compute_event_reference_hash(event, hash_algorithm=hashlib.sha256):
	102	def compute_event_reference_hash(
	103	event, hash_algorithm: Hasher = hashlib.sha256
	104	) -> Tuple[str, bytes]:
97	105	"""Computes the event reference hash. This is the hash of the redacted
98	106	event.
99	107
100	108	Args:
101		event (FrozenEvent)
	109	event
102	110	hash_algorithm: A hasher from `hashlib`, e.g. hashlib.sha256, to use
103	111	to hash the event
104	112
105	113	Returns:
106		tuple[str, bytes]: A tuple of the name of hash and the hash as raw
107		bytes.
	114	A tuple of the name of hash and the hash as raw bytes.
108	115	"""
109	116	tmp_event = prune_event(event)
110	117	event_dict = tmp_event.get_pdu_json()

155	162	event_dict: JsonDict,
156	163	signature_name: str,
157	164	signing_key: SigningKey,
158		):
	165	) -> None:
159	166	"""Add content hash and sign the event
160	167
161	168	Args:

+138

-104

synapse/crypto/keyring.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
	16	import abc
16	17	import logging
17	18	import urllib
18	19	from collections import defaultdict
	20	from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Set, Tuple
19	21
20	22	import attr
21	23	from signedjson.key import (

39	41	RequestSendFailed,
40	42	SynapseError,
41	43	)
	44	from synapse.config.key import TrustedKeyServer
42	45	from synapse.logging.context import (
43	46	PreserveLoggingContext,
44	47	make_deferred_yieldable,

46	49	run_in_background,
47	50	)
48	51	from synapse.storage.keys import FetchKeyResult
	52	from synapse.types import JsonDict
49	53	from synapse.util import unwrapFirstError
50	54	from synapse.util.async_helpers import yieldable_gather_results
51	55	from synapse.util.metrics import Measure
52	56	from synapse.util.retryutils import NotRetryingDestination
53	57
	58	if TYPE_CHECKING:
	59	from synapse.app.homeserver import HomeServer
	60
54	61	logger = logging.getLogger(__name__)
55	62
56	63

60	67	A request to verify a JSON object.
61	68
62	69	Attributes:
63		server_name(str): The name of the server to verify against.
64
65		key_ids(set[str]): The set of key_ids to that could be used to verify the
66		JSON object
67
68		json_object(dict): The JSON object to verify.
69
70		minimum_valid_until_ts (int): time at which we require the signing key to
	70	server_name: The name of the server to verify against.
	71
	72	json_object: The JSON object to verify.
	73
	74	minimum_valid_until_ts: time at which we require the signing key to
71	75	be valid. (0 implies we don't care)
	76
	77	request_name: The name of the request.
	78
	79	key_ids: The set of key_ids to that could be used to verify the JSON object
72	80
73	81	key_ready (Deferred[str, str, nacl.signing.VerifyKey]):
74	82	A deferred (server_name, key_id, verify_key) tuple that resolves when

79	87	errbacks with an M_UNAUTHORIZED SynapseError.
80	88	"""
81	89
82		server_name = attr.ib()
83		json_object = attr.ib()
84		minimum_valid_until_ts = attr.ib()
85		request_name = attr.ib()
86		key_ids = attr.ib(init=False)
87		key_ready = attr.ib(default=attr.Factory(defer.Deferred))
	90	server_name = attr.ib(type=str)
	91	json_object = attr.ib(type=JsonDict)
	92	minimum_valid_until_ts = attr.ib(type=int)
	93	request_name = attr.ib(type=str)
	94	key_ids = attr.ib(init=False, type=List[str])
	95	key_ready = attr.ib(default=attr.Factory(defer.Deferred), type=defer.Deferred)
88	96
89	97	def __attrs_post_init__(self):
90	98	self.key_ids = signature_ids(self.json_object, self.server_name)

95	103
96	104
97	105	class Keyring:
98		def __init__(self, hs, key_fetchers=None):
	106	def __init__(
	107	self, hs: "HomeServer", key_fetchers: "Optional[Iterable[KeyFetcher]]" = None
	108	):
99	109	self.clock = hs.get_clock()
100	110
101	111	if key_fetchers is None:

111	121	# completes.
112	122	#
113	123	# These are regular, logcontext-agnostic Deferreds.
114		self.key_downloads = {}
	124	self.key_downloads = {} # type: Dict[str, defer.Deferred]
115	125
116	126	def verify_json_for_server(
117		self, server_name, json_object, validity_time, request_name
118		):
	127	self,
	128	server_name: str,
	129	json_object: JsonDict,
	130	validity_time: int,
	131	request_name: str,
	132	) -> defer.Deferred:
119	133	"""Verify that a JSON object has been signed by a given server
120	134
121	135	Args:
122		server_name (str): name of the server which must have signed this object
123
124		json_object (dict): object to be checked
125
126		validity_time (int): timestamp at which we require the signing key to
	136	server_name: name of the server which must have signed this object
	137
	138	json_object: object to be checked
	139
	140	validity_time: timestamp at which we require the signing key to
127	141	be valid. (0 implies we don't care)
128	142
129		request_name (str): an identifier for this json object (eg, an event id)
	143	request_name: an identifier for this json object (eg, an event id)
130	144	for logging.
131	145
132	146	Returns:

137	151	requests = (req,)
138	152	return make_deferred_yieldable(self._verify_objects(requests)[0])
139	153
140		def verify_json_objects_for_server(self, server_and_json):
	154	def verify_json_objects_for_server(
	155	self, server_and_json: Iterable[Tuple[str, dict, int, str]]
	156	) -> List[defer.Deferred]:
141	157	"""Bulk verifies signatures of json objects, bulk fetching keys as
142	158	necessary.
143	159
144	160	Args:
145		server_and_json (iterable[Tuple[str, dict, int, str]):
	161	server_and_json:
146	162	Iterable of (server_name, json_object, validity_time, request_name)
147	163	tuples.
148	164

163	179	for server_name, json_object, validity_time, request_name in server_and_json
164	180	)
165	181
166		def _verify_objects(self, verify_requests):
	182	def _verify_objects(
	183	self, verify_requests: Iterable[VerifyJsonRequest]
	184	) -> List[defer.Deferred]:
167	185	"""Does the work of verify_json_[objects_]for_server
168	186
169	187
170	188	Args:
171		verify_requests (iterable[VerifyJsonRequest]):
172		Iterable of verification requests.
	189	verify_requests: Iterable of verification requests.
173	190
174	191	Returns:
175	192	List<Deferred[None]>: for each input item, a deferred indicating success

181	198	key_lookups = []
182	199	handle = preserve_fn(_handle_key_deferred)
183	200
184		def process(verify_request):
	201	def process(verify_request: VerifyJsonRequest) -> defer.Deferred:
185	202	"""Process an entry in the request list
186	203
187	204	Adds a key request to key_lookups, and returns a deferred which

221	238
222	239	return results
223	240
224		async def _start_key_lookups(self, verify_requests):
	241	async def _start_key_lookups(
	242	self, verify_requests: List[VerifyJsonRequest]
	243	) -> None:
225	244	"""Sets off the key fetches for each verify request
226	245
227	246	Once each fetch completes, verify_request.key_ready will be resolved.
228	247
229	248	Args:
230		verify_requests (List[VerifyJsonRequest]):
	249	verify_requests:
231	250	"""
232	251
233	252	try:
234	253	# map from server name to a set of outstanding request ids
235		server_to_request_ids = {}
	254	server_to_request_ids = {} # type: Dict[str, Set[int]]
236	255
237	256	for verify_request in verify_requests:
238	257	server_name = verify_request.server_name

274	293	except Exception:
275	294	logger.exception("Error starting key lookups")
276	295
277		async def wait_for_previous_lookups(self, server_names) -> None:
	296	async def wait_for_previous_lookups(self, server_names: Iterable[str]) -> None:
278	297	"""Waits for any previous key lookups for the given servers to finish.
279	298
280	299	Args:
281		server_names (Iterable[str]): list of servers which we want to look up
	300	server_names: list of servers which we want to look up
282	301
283	302	Returns:
284	303	Resolves once all key lookups for the given servers have

303	322
304	323	loop_count += 1
305	324
306		def _get_server_verify_keys(self, verify_requests):
	325	def _get_server_verify_keys(self, verify_requests: List[VerifyJsonRequest]) -> None:
307	326	"""Tries to find at least one key for each verify request
308	327
309	328	For each verify_request, verify_request.key_ready is called back with

311	330	with a SynapseError if none of the keys are found.
312	331
313	332	Args:
314		verify_requests (list[VerifyJsonRequest]): list of verify requests
	333	verify_requests: list of verify requests
315	334	"""
316	335
317	336	remaining_requests = {rq for rq in verify_requests if not rq.key_ready.called}

365	384
366	385	run_in_background(do_iterations)
367	386
368		async def _attempt_key_fetches_with_fetcher(self, fetcher, remaining_requests):
	387	async def _attempt_key_fetches_with_fetcher(
	388	self, fetcher: "KeyFetcher", remaining_requests: Set[VerifyJsonRequest]
	389	):
369	390	"""Use a key fetcher to attempt to satisfy some key requests
370	391
371	392	Args:
372		fetcher (KeyFetcher): fetcher to use to fetch the keys
373		remaining_requests (set[VerifyJsonRequest]): outstanding key requests.
	393	fetcher: fetcher to use to fetch the keys
	394	remaining_requests: outstanding key requests.
374	395	Any successfully-completed requests will be removed from the list.
375	396	"""
376		# dict[str, dict[str, int]]: keys to fetch.
	397	# The keys to fetch.
377	398	# server_name -> key_id -> min_valid_ts
378		missing_keys = defaultdict(dict)
	399	missing_keys = defaultdict(dict) # type: Dict[str, Dict[str, int]]
379	400
380	401	for verify_request in remaining_requests:
381	402	# any completed requests should already have been removed

437	458	remaining_requests.difference_update(completed)
438	459
439	460
440		class KeyFetcher:
441		async def get_keys(self, keys_to_fetch):
442		"""
443		Args:
444		keys_to_fetch (dict[str, dict[str, int]]):
	461	class KeyFetcher(metaclass=abc.ABCMeta):
	462	@abc.abstractmethod
	463	async def get_keys(
	464	self, keys_to_fetch: Dict[str, Dict[str, int]]
	465	) -> Dict[str, Dict[str, FetchKeyResult]]:
	466	"""
	467	Args:
	468	keys_to_fetch:
445	469	the keys to be fetched. server_name -> key_id -> min_valid_ts
446	470
447	471	Returns:
448		Deferred[dict[str, dict[str, synapse.storage.keys.FetchKeyResult\|None]]]:
449		map from server_name -> key_id -> FetchKeyResult
	472	Map from server_name -> key_id -> FetchKeyResult
450	473	"""
451	474	raise NotImplementedError
452	475

454	477	class StoreKeyFetcher(KeyFetcher):
455	478	"""KeyFetcher impl which fetches keys from our data store"""
456	479
457		def __init__(self, hs):
	480	def __init__(self, hs: "HomeServer"):
458	481	self.store = hs.get_datastore()
459	482
460		async def get_keys(self, keys_to_fetch):
	483	async def get_keys(
	484	self, keys_to_fetch: Dict[str, Dict[str, int]]
	485	) -> Dict[str, Dict[str, FetchKeyResult]]:
461	486	"""see KeyFetcher.get_keys"""
462	487
463		keys_to_fetch = (
	488	key_ids_to_fetch = (
464	489	(server_name, key_id)
465	490	for server_name, keys_for_server in keys_to_fetch.items()
466	491	for key_id in keys_for_server.keys()
467	492	)
468	493
469		res = await self.store.get_server_verify_keys(keys_to_fetch)
470		keys = {}
	494	res = await self.store.get_server_verify_keys(key_ids_to_fetch)
	495	keys = {} # type: Dict[str, Dict[str, FetchKeyResult]]
471	496	for (server_name, key_id), key in res.items():
472	497	keys.setdefault(server_name, {})[key_id] = key
473	498	return keys
474	499
475	500
476		class BaseV2KeyFetcher:
477		def __init__(self, hs):
	501	class BaseV2KeyFetcher(KeyFetcher):
	502	def __init__(self, hs: "HomeServer"):
478	503	self.store = hs.get_datastore()
479	504	self.config = hs.get_config()
480	505
481		async def process_v2_response(self, from_server, response_json, time_added_ms):
	506	async def process_v2_response(
	507	self, from_server: str, response_json: JsonDict, time_added_ms: int
	508	) -> Dict[str, FetchKeyResult]:
482	509	"""Parse a 'Server Keys' structure from the result of a /key request
483	510
484	511	This is used to parse either the entirety of the response from

492	519	to /_matrix/key/v2/query.
493	520
494	521	Args:
495		from_server (str): the name of the server producing this result: either
	522	from_server: the name of the server producing this result: either
496	523	the origin server for a /_matrix/key/v2/server request, or the notary
497	524	for a /_matrix/key/v2/query.
498	525
499		response_json (dict): the json-decoded Server Keys response object
500
501		time_added_ms (int): the timestamp to record in server_keys_json
	526	response_json: the json-decoded Server Keys response object
	527
	528	time_added_ms: the timestamp to record in server_keys_json
502	529
503	530	Returns:
504		Deferred[dict[str, FetchKeyResult]]: map from key_id to result object
	531	Map from key_id to result object
505	532	"""
506	533	ts_valid_until_ms = response_json["valid_until_ts"]
507	534

574	601	class PerspectivesKeyFetcher(BaseV2KeyFetcher):
575	602	"""KeyFetcher impl which fetches keys from the "perspectives" servers"""
576	603
577		def __init__(self, hs):
	604	def __init__(self, hs: "HomeServer"):
578	605	super().__init__(hs)
579	606	self.clock = hs.get_clock()
580		self.client = hs.get_http_client()
	607	self.client = hs.get_federation_http_client()
581	608	self.key_servers = self.config.key_servers
582	609
583		async def get_keys(self, keys_to_fetch):
	610	async def get_keys(
	611	self, keys_to_fetch: Dict[str, Dict[str, int]]
	612	) -> Dict[str, Dict[str, FetchKeyResult]]:
584	613	"""see KeyFetcher.get_keys"""
585	614
586		async def get_key(key_server):
	615	async def get_key(key_server: TrustedKeyServer) -> Dict:
587	616	try:
588		result = await self.get_server_verify_key_v2_indirect(
	617	return await self.get_server_verify_key_v2_indirect(
589	618	keys_to_fetch, key_server
590	619	)
591		return result
592	620	except KeyLookupError as e:
593	621	logger.warning(
594	622	"Key lookup failed from %r: %s", key_server.server_name, e

610	638	).addErrback(unwrapFirstError)
611	639	)
612	640
613		union_of_keys = {}
	641	union_of_keys = {} # type: Dict[str, Dict[str, FetchKeyResult]]
614	642	for result in results:
615	643	for server_name, keys in result.items():
616	644	union_of_keys.setdefault(server_name, {}).update(keys)
617	645
618	646	return union_of_keys
619	647
620		async def get_server_verify_key_v2_indirect(self, keys_to_fetch, key_server):
621		"""
622		Args:
623		keys_to_fetch (dict[str, dict[str, int]]):
	648	async def get_server_verify_key_v2_indirect(
	649	self, keys_to_fetch: Dict[str, Dict[str, int]], key_server: TrustedKeyServer
	650	) -> Dict[str, Dict[str, FetchKeyResult]]:
	651	"""
	652	Args:
	653	keys_to_fetch:
624	654	the keys to be fetched. server_name -> key_id -> min_valid_ts
625	655
626		key_server (synapse.config.key.TrustedKeyServer): notary server to query for
627		the keys
	656	key_server: notary server to query for the keys
628	657
629	658	Returns:
630		dict[str, dict[str, synapse.storage.keys.FetchKeyResult]]: map
631		from server_name -> key_id -> FetchKeyResult
	659	Map from server_name -> key_id -> FetchKeyResult
632	660
633	661	Raises:
634	662	KeyLookupError if there was an error processing the entire response from

661	689	except HttpResponseException as e:
662	690	raise KeyLookupError("Remote server returned an error: %s" % (e,))
663	691
664		keys = {}
665		added_keys = []
	692	keys = {} # type: Dict[str, Dict[str, FetchKeyResult]]
	693	added_keys = [] # type: List[Tuple[str, str, FetchKeyResult]]
666	694
667	695	time_now_ms = self.clock.time_msec()
668	696
	697	assert isinstance(query_response, dict)
669	698	for response in query_response["server_keys"]:
670	699	# do this first, so that we can give useful errors thereafter
671	700	server_name = response.get("server_name")

703	732
704	733	return keys
705	734
706		def _validate_perspectives_response(self, key_server, response):
	735	def _validate_perspectives_response(
	736	self, key_server: TrustedKeyServer, response: JsonDict
	737	) -> None:
707	738	"""Optionally check the signature on the result of a /key/query request
708	739
709	740	Args:
710		key_server (synapse.config.key.TrustedKeyServer): the notary server that
711		produced this result
712
713		response (dict): the json-decoded Server Keys response object
	741	key_server: the notary server that produced this result
	742
	743	response: the json-decoded Server Keys response object
714	744	"""
715	745	perspective_name = key_server.server_name
716	746	perspective_keys = key_server.verify_keys

744	774	class ServerKeyFetcher(BaseV2KeyFetcher):
745	775	"""KeyFetcher impl which fetches keys from the origin servers"""
746	776
747		def __init__(self, hs):
	777	def __init__(self, hs: "HomeServer"):
748	778	super().__init__(hs)
749	779	self.clock = hs.get_clock()
750		self.client = hs.get_http_client()
751
752		async def get_keys(self, keys_to_fetch):
753		"""
754		Args:
755		keys_to_fetch (dict[str, iterable[str]]):
	780	self.client = hs.get_federation_http_client()
	781
	782	async def get_keys(
	783	self, keys_to_fetch: Dict[str, Dict[str, int]]
	784	) -> Dict[str, Dict[str, FetchKeyResult]]:
	785	"""
	786	Args:
	787	keys_to_fetch:
756	788	the keys to be fetched. server_name -> key_ids
757	789
758	790	Returns:
759		dict[str, dict[str, synapse.storage.keys.FetchKeyResult\|None]]:
760		map from server_name -> key_id -> FetchKeyResult
	791	Map from server_name -> key_id -> FetchKeyResult
761	792	"""
762	793
763	794	results = {}
764	795
765		async def get_key(key_to_fetch_item):
	796	async def get_key(key_to_fetch_item: Tuple[str, Dict[str, int]]) -> None:
766	797	server_name, key_ids = key_to_fetch_item
767	798	try:
768	799	keys = await self.get_server_verify_key_v2_direct(server_name, key_ids)

777	808	await yieldable_gather_results(get_key, keys_to_fetch.items())
778	809	return results
779	810
780		async def get_server_verify_key_v2_direct(self, server_name, key_ids):
781		"""
782
783		Args:
784		server_name (str):
785		key_ids (iterable[str]):
	811	async def get_server_verify_key_v2_direct(
	812	self, server_name: str, key_ids: Iterable[str]
	813	) -> Dict[str, FetchKeyResult]:
	814	"""
	815
	816	Args:
	817	server_name:
	818	key_ids:
786	819
787	820	Returns:
788		dict[str, FetchKeyResult]: map from key ID to lookup result
	821	Map from key ID to lookup result
789	822
790	823	Raises:
791	824	KeyLookupError if there was a problem making the lookup
792	825	"""
793		keys = {} # type: dict[str, FetchKeyResult]
	826	keys = {} # type: Dict[str, FetchKeyResult]
794	827
795	828	for requested_key_id in key_ids:
796	829	# we may have found this key as a side-effect of asking for another.

824	857	except HttpResponseException as e:
825	858	raise KeyLookupError("Remote server returned an error: %s" % (e,))
826	859
	860	assert isinstance(response, dict)
827	861	if response["server_name"] != server_name:
828	862	raise KeyLookupError(
829	863	"Expected a response for server %r not %r"

845	879	return keys
846	880
847	881
848		async def _handle_key_deferred(verify_request) -> None:
	882	async def _handle_key_deferred(verify_request: VerifyJsonRequest) -> None:
849	883	"""Waits for the key to become available, and then performs a verification
850	884
851	885	Args:
852		verify_request (VerifyJsonRequest):
	886	verify_request:
853	887
854	888	Raises:
855	889	SynapseError if there was a problem performing the verification

+59

-36

synapse/events/spamcheck.py less more

14	14	# limitations under the License.
15	15
16	16	import inspect
17		from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
	17	from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
18	18
19	19	from synapse.spam_checker_api import RegistrationBehaviour
20	20	from synapse.types import Collection
	21	from synapse.util.async_helpers import maybe_awaitable
21	22
22	23	if TYPE_CHECKING:
23	24	import synapse.events

38	39	else:
39	40	self.spam_checkers.append(module(config=config))
40	41
41		def check_event_for_spam(self, event: "synapse.events.EventBase") -> bool:
	42	async def check_event_for_spam(
	43	self, event: "synapse.events.EventBase"
	44	) -> Union[bool, str]:
42	45	"""Checks if a given event is considered "spammy" by this server.
43	46
44	47	If the server considers an event spammy, then it will be rejected if

49	52	event: the event to be checked
50	53
51	54	Returns:
52		True if the event is spammy.
53		"""
54		for spam_checker in self.spam_checkers:
55		if spam_checker.check_event_for_spam(event):
	55	True or a string if the event is spammy. If a string is returned it
	56	will be used as the error message returned to the user.
	57	"""
	58	for spam_checker in self.spam_checkers:
	59	if await maybe_awaitable(spam_checker.check_event_for_spam(event)):
56	60	return True
57	61
58	62	return False
59	63
60		def user_may_invite(
	64	async def user_may_invite(
61	65	self, inviter_userid: str, invitee_userid: str, room_id: str
62	66	) -> bool:
63	67	"""Checks if a given user may send an invite

74	78	"""
75	79	for spam_checker in self.spam_checkers:
76	80	if (
77		spam_checker.user_may_invite(inviter_userid, invitee_userid, room_id)
78		is False
79		):
80		return False
81
82		return True
83
84		def user_may_create_room(self, userid: str) -> bool:
	81	await maybe_awaitable(
	82	spam_checker.user_may_invite(
	83	inviter_userid, invitee_userid, room_id
	84	)
	85	)
	86	is False
	87	):
	88	return False
	89
	90	return True
	91
	92	async def user_may_create_room(self, userid: str) -> bool:
85	93	"""Checks if a given user may create a room
86	94
87	95	If this method returns false, the creation request will be rejected.

93	101	True if the user may create a room, otherwise False
94	102	"""
95	103	for spam_checker in self.spam_checkers:
96		if spam_checker.user_may_create_room(userid) is False:
97		return False
98
99		return True
100
101		def user_may_create_room_alias(self, userid: str, room_alias: str) -> bool:
	104	if (
	105	await maybe_awaitable(spam_checker.user_may_create_room(userid))
	106	is False
	107	):
	108	return False
	109
	110	return True
	111
	112	async def user_may_create_room_alias(self, userid: str, room_alias: str) -> bool:
102	113	"""Checks if a given user may create a room alias
103	114
104	115	If this method returns false, the association request will be rejected.

111	122	True if the user may create a room alias, otherwise False
112	123	"""
113	124	for spam_checker in self.spam_checkers:
114		if spam_checker.user_may_create_room_alias(userid, room_alias) is False:
115		return False
116
117		return True
118
119		def user_may_publish_room(self, userid: str, room_id: str) -> bool:
	125	if (
	126	await maybe_awaitable(
	127	spam_checker.user_may_create_room_alias(userid, room_alias)
	128	)
	129	is False
	130	):
	131	return False
	132
	133	return True
	134
	135	async def user_may_publish_room(self, userid: str, room_id: str) -> bool:
120	136	"""Checks if a given user may publish a room to the directory
121	137
122	138	If this method returns false, the publish request will be rejected.

129	145	True if the user may publish the room, otherwise False
130	146	"""
131	147	for spam_checker in self.spam_checkers:
132		if spam_checker.user_may_publish_room(userid, room_id) is False:
133		return False
134
135		return True
136
137		def check_username_for_spam(self, user_profile: Dict[str, str]) -> bool:
	148	if (
	149	await maybe_awaitable(
	150	spam_checker.user_may_publish_room(userid, room_id)
	151	)
	152	is False
	153	):
	154	return False
	155
	156	return True
	157
	158	async def check_username_for_spam(self, user_profile: Dict[str, str]) -> bool:
138	159	"""Checks if a user ID or display name are considered "spammy" by this server.
139	160
140	161	If the server considers a username spammy, then it will not be included in

156	177	if checker:
157	178	# Make a copy of the user profile object to ensure the spam checker
158	179	# cannot modify it.
159		if checker(user_profile.copy()):
	180	if await maybe_awaitable(checker(user_profile.copy())):
160	181	return True
161	182
162	183	return False
163	184
164		def check_registration_for_spam(
	185	async def check_registration_for_spam(
165	186	self,
166	187	email_threepid: Optional[dict],
167	188	username: Optional[str],

184	205	# spam checker
185	206	checker = getattr(spam_checker, "check_registration_for_spam", None)
186	207	if checker:
187		behaviour = checker(email_threepid, username, request_info)
	208	behaviour = await maybe_awaitable(
	209	checker(email_threepid, username, request_info)
	210	)
188	211	assert isinstance(behaviour, RegistrationBehaviour)
189	212	if behaviour != RegistrationBehaviour.ALLOW:
190	213	return behaviour

+6

-1

synapse/federation/federation_base.py less more

77	77
78	78	ctx = current_context()
79	79
	80	@defer.inlineCallbacks
80	81	def callback(_, pdu: EventBase):
81	82	with PreserveLoggingContext(ctx):
82	83	if not check_event_content_hash(pdu):

104	105	)
105	106	return redacted_event
106	107
107		if self.spam_checker.check_event_for_spam(pdu):
	108	result = yield defer.ensureDeferred(
	109	self.spam_checker.check_event_for_spam(pdu)
	110	)
	111
	112	if result:
108	113	logger.warning(
109	114	"Event contains spam, redacting %s: %s",
110	115	pdu.event_id,

+0

-1

synapse/federation/federation_server.py less more

844	844
845	845	def __init__(self, hs: "HomeServer"):
846	846	self.config = hs.config
847		self.http_client = hs.get_simple_http_client()
848	847	self.clock = hs.get_clock()
849	848	self._instance_name = hs.get_instance_name()
850	849

+1

-1

synapse/federation/transport/client.py less more

34	34
35	35	def __init__(self, hs):
36	36	self.server_name = hs.hostname
37		self.client = hs.get_http_client()
	37	self.client = hs.get_federation_http_client()
38	38
39	39	@log_function
40	40	def get_room_state_ids(self, destination, room_id, event_id):

+2

-2

synapse/federation/transport/server.py less more

143	143	):
144	144	raise FederationDeniedError(origin)
145	145
146		if not json_request["signatures"]:
	146	if origin is None or not json_request["signatures"]:
147	147	raise NoAuthenticationError(
148	148	401, "Missing Authorization headers", Codes.UNAUTHORIZED
149	149	)

1461	1461
1462	1462	Args:
1463	1463	hs (synapse.server.HomeServer): homeserver
1464		resource (TransportLayerServer): resource class to register to
	1464	resource (JsonResource): resource class to register to
1465	1465	authenticator (Authenticator): authenticator to use
1466	1466	ratelimiter (util.ratelimitutils.FederationRateLimiter): ratelimiter to use
1467	1467	servlet_groups (list[str], optional): List of servlet groups to register.

+4

-0

synapse/handlers/_base.py less more

31	31	class BaseHandler:
32	32	"""
33	33	Common base class for the event handlers.
	34
	35	Deprecated: new code should not use this. Instead, Handler classes should define the
	36	fields they actually need. The utility methods should either be factored out to
	37	standalone helper functions, or to different Handler classes.
34	38	"""
35	39
36	40	def __init__(self, hs: "HomeServer"):

+42

-29

synapse/handlers/admin.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	import abc
15	16	import logging
16		from typing import List
	17	from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
17	18
18	19	from synapse.api.constants import Membership
19		from synapse.events import FrozenEvent
20		from synapse.types import RoomStreamToken, StateMap
	20	from synapse.events import EventBase
	21	from synapse.types import JsonDict, RoomStreamToken, StateMap, UserID
21	22	from synapse.visibility import filter_events_for_client
22	23
23	24	from ._base import BaseHandler
24	25
	26	if TYPE_CHECKING:
	27	from synapse.app.homeserver import HomeServer
	28
25	29	logger = logging.getLogger(__name__)
26	30
27	31
28	32	class AdminHandler(BaseHandler):
29		def __init__(self, hs):
	33	def __init__(self, hs: "HomeServer"):
30	34	super().__init__(hs)
31	35
32	36	self.storage = hs.get_storage()
33	37	self.state_store = self.storage.state
34	38
35		async def get_whois(self, user):
	39	async def get_whois(self, user: UserID) -> JsonDict:
36	40	connections = []
37	41
38	42	sessions = await self.store.get_user_ip_and_agents(user)

52	56
53	57	return ret
54	58
55		async def get_user(self, user):
	59	async def get_user(self, user: UserID) -> Optional[JsonDict]:
56	60	"""Function to get user details"""
57	61	ret = await self.store.get_user_by_id(user.to_string())
58	62	if ret:

63	67	ret["threepids"] = threepids
64	68	return ret
65	69
66		async def export_user_data(self, user_id, writer):
	70	async def export_user_data(self, user_id: str, writer: "ExfiltrationWriter") -> Any:
67	71	"""Write all data we have on the user to the given writer.
68	72
69	73	Args:
70		user_id (str)
71		writer (ExfiltrationWriter)
	74	user_id: The user ID to fetch data of.
	75	writer: The writer to write to.
72	76
73	77	Returns:
74	78	Resolves when all data for a user has been written.

127	131	from_key = RoomStreamToken(0, 0)
128	132	to_key = RoomStreamToken(None, stream_ordering)
129	133
130		written_events = set() # Events that we've processed in this room
	134	# Events that we've processed in this room
	135	written_events = set() # type: Set[str]
131	136
132	137	# We need to track gaps in the events stream so that we can then
133	138	# write out the state at those events. We do this by keeping track

139	144
140	145	# The reverse mapping to above, i.e. map from unseen event to events
141	146	# that have the unseen event in their prev_events, i.e. the unseen
142		# events "children". dict[str, set[str]]
143		unseen_to_child_events = {}
	147	# events "children".
	148	unseen_to_child_events = {} # type: Dict[str, Set[str]]
144	149
145	150	# We fetch events in the room the user could see by fetching all
146	151	# events that we have and then filtering, this isn't the most

196	201	return writer.finished()
197	202
198	203
199		class ExfiltrationWriter:
	204	class ExfiltrationWriter(metaclass=abc.ABCMeta):
200	205	"""Interface used to specify how to write exported data.
201	206	"""
202	207
203		def write_events(self, room_id: str, events: List[FrozenEvent]):
	208	@abc.abstractmethod
	209	def write_events(self, room_id: str, events: List[EventBase]) -> None:
204	210	"""Write a batch of events for a room.
205	211	"""
206		pass
207
208		def write_state(self, room_id: str, event_id: str, state: StateMap[FrozenEvent]):
	212	raise NotImplementedError()
	213
	214	@abc.abstractmethod
	215	def write_state(
	216	self, room_id: str, event_id: str, state: StateMap[EventBase]
	217	) -> None:
209	218	"""Write the state at the given event in the room.
210	219
211	220	This only gets called for backward extremities rather than for each
212	221	event.
213	222	"""
214		pass
215
216		def write_invite(self, room_id: str, event: FrozenEvent, state: StateMap[dict]):
	223	raise NotImplementedError()
	224
	225	@abc.abstractmethod
	226	def write_invite(
	227	self, room_id: str, event: EventBase, state: StateMap[dict]
	228	) -> None:
217	229	"""Write an invite for the room, with associated invite state.
218	230
219	231	Args:
220		room_id
221		event
222		state: A subset of the state at the
223		invite, with a subset of the event keys (type, state_key
224		content and sender)
225		"""
226
227		def finished(self):
	232	room_id: The room ID the invite is for.
	233	event: The invite event.
	234	state: A subset of the state at the invite, with a subset of the
	235	event keys (type, state_key content and sender).
	236	"""
	237	raise NotImplementedError()
	238
	239	@abc.abstractmethod
	240	def finished(self) -> Any:
228	241	"""Called when all data has successfully been exported and written.
229	242
230	243	This functions return value is passed to the caller of
231	244	`export_user_data`.
232	245	"""
233		pass
	246	raise NotImplementedError()

+100

-50

synapse/handlers/auth.py less more

13	13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	14	# See the License for the specific language governing permissions and
15	15	# limitations under the License.
16		import inspect
17	16	import logging
18	17	import time
19	18	import unicodedata

21	20	from typing import (
22	21	TYPE_CHECKING,
23	22	Any,
	23	Awaitable,
24	24	Callable,
25	25	Dict,
26	26	Iterable,

34	34	import attr
35	35	import bcrypt
36	36	import pymacaroons
	37
	38	from twisted.web.http import Request
37	39
38	40	from synapse.api.constants import LoginType
39	41	from synapse.api.errors import (

55	57	from synapse.module_api import ModuleApi
56	58	from synapse.types import JsonDict, Requester, UserID
57	59	from synapse.util import stringutils as stringutils
	60	from synapse.util.async_helpers import maybe_awaitable
58	61	from synapse.util.msisdn import phone_number_to_msisdn
59	62	from synapse.util.threepids import canonicalise_email
60	63

192	195	self.hs = hs # FIXME better possibility to access registrationHandler later?
193	196	self.macaroon_gen = hs.get_macaroon_generator()
194	197	self._password_enabled = hs.config.password_enabled
195		self._sso_enabled = (
196		hs.config.cas_enabled or hs.config.saml2_enabled or hs.config.oidc_enabled
197		)
198
199		# we keep this as a list despite the O(N^2) implication so that we can
200		# keep PASSWORD first and avoid confusing clients which pick the first
201		# type in the list. (NB that the spec doesn't require us to do so and
202		# clients which favour types that they don't understand over those that
203		# they do are technically broken)
	198	self._password_localdb_enabled = hs.config.password_localdb_enabled
204	199
205	200	# start out by assuming PASSWORD is enabled; we will remove it later if not.
206		login_types = []
207		if hs.config.password_localdb_enabled:
208		login_types.append(LoginType.PASSWORD)
	201	login_types = set()
	202	if self._password_localdb_enabled:
	203	login_types.add(LoginType.PASSWORD)
209	204
210	205	for provider in self.password_providers:
211		if hasattr(provider, "get_supported_login_types"):
212		for t in provider.get_supported_login_types().keys():
213		if t not in login_types:
214		login_types.append(t)
	206	login_types.update(provider.get_supported_login_types().keys())
215	207
216	208	if not self._password_enabled:
	209	login_types.discard(LoginType.PASSWORD)
	210
	211	# Some clients just pick the first type in the list. In this case, we want
	212	# them to use PASSWORD (rather than token or whatever), so we want to make sure
	213	# that comes first, where it's present.
	214	self._supported_login_types = []
	215	if LoginType.PASSWORD in login_types:
	216	self._supported_login_types.append(LoginType.PASSWORD)
217	217	login_types.remove(LoginType.PASSWORD)
218
219		self._supported_login_types = login_types
220
221		# Login types and UI Auth types have a heavy overlap, but are not
222		# necessarily identical. Login types have SSO (and other login types)
223		# added in the rest layer, see synapse.rest.client.v1.login.LoginRestServerlet.on_GET.
224		ui_auth_types = login_types.copy()
225		if self._sso_enabled:
226		ui_auth_types.append(LoginType.SSO)
227		self._supported_ui_auth_types = ui_auth_types
	218	self._supported_login_types.extend(login_types)
228	219
229	220	# Ratelimiter for failed auth during UIA. Uses same ratelimit config
230	221	# as per `rc_login.failed_attempts`.

233	224	rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
234	225	burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
235	226	)
	227
	228	# The number of seconds to keep a UI auth session active.
	229	self._ui_auth_session_timeout = hs.config.ui_auth_session_timeout
236	230
237	231	# Ratelimitier for failed /login attempts
238	232	self._failed_login_attempts_ratelimiter = Ratelimiter(

291	285	request_body: Dict[str, Any],
292	286	clientip: str,
293	287	description: str,
294		) -> Tuple[dict, str]:
	288	) -> Tuple[dict, Optional[str]]:
295	289	"""
296	290	Checks that the user is who they claim to be, via a UI auth.
297	291

318	312	have been given only in a previous call).
319	313
320	314	'session_id' is the ID of this session, either passed in by the
321		client or assigned by this call
	315	client or assigned by this call. This is None if UI auth was
	316	skipped (by re-using a previous validation).
322	317
323	318	Raises:
324	319	InteractiveAuthIncompleteError if the client has not yet completed

332	327
333	328	"""
334	329
	330	if self._ui_auth_session_timeout:
	331	last_validated = await self.store.get_access_token_last_validated(
	332	requester.access_token_id
	333	)
	334	if self.clock.time_msec() - last_validated < self._ui_auth_session_timeout:
	335	# Return the input parameters, minus the auth key, which matches
	336	# the logic in check_ui_auth.
	337	request_body.pop("auth", None)
	338	return request_body, None
	339
335	340	user_id = requester.user.to_string()
336	341
337	342	# Check if we should be ratelimited due to too many previous failed attempts
338	343	self._failed_uia_attempts_ratelimiter.ratelimit(user_id, update=False)
339	344
340	345	# build a list of supported flows
341		flows = [[login_type] for login_type in self._supported_ui_auth_types]
	346	supported_ui_auth_types = await self._get_available_ui_auth_types(
	347	requester.user
	348	)
	349	flows = [[login_type] for login_type in supported_ui_auth_types]
342	350
343	351	try:
344	352	result, params, session_id = await self.check_ui_auth(

350	358	raise
351	359
352	360	# find the completed login type
353		for login_type in self._supported_ui_auth_types:
	361	for login_type in supported_ui_auth_types:
354	362	if login_type not in result:
355	363	continue
356	364

364	372	if user_id != requester.user.to_string():
365	373	raise AuthError(403, "Invalid auth")
366	374
	375	# Note that the access token has been validated.
	376	await self.store.update_access_token_last_validated(requester.access_token_id)
	377
367	378	return params, session_id
	379
	380	async def _get_available_ui_auth_types(self, user: UserID) -> Iterable[str]:
	381	"""Get a list of the authentication types this user can use
	382	"""
	383
	384	ui_auth_types = set()
	385
	386	# if the HS supports password auth, and the user has a non-null password, we
	387	# support password auth
	388	if self._password_localdb_enabled and self._password_enabled:
	389	lookupres = await self._find_user_id_and_pwd_hash(user.to_string())
	390	if lookupres:
	391	_, password_hash = lookupres
	392	if password_hash:
	393	ui_auth_types.add(LoginType.PASSWORD)
	394
	395	# also allow auth from password providers
	396	for provider in self.password_providers:
	397	for t in provider.get_supported_login_types().keys():
	398	if t == LoginType.PASSWORD and not self._password_enabled:
	399	continue
	400	ui_auth_types.add(t)
	401
	402	# if sso is enabled, allow the user to log in via SSO iff they have a mapping
	403	# from sso to mxid.
	404	if self.hs.config.saml2.saml2_enabled or self.hs.config.oidc.oidc_enabled:
	405	if await self.store.get_external_ids_by_user(user.to_string()):
	406	ui_auth_types.add(LoginType.SSO)
	407
	408	# Our CAS impl does not (yet) correctly register users in user_external_ids,
	409	# so always offer that if it's available.
	410	if self.hs.config.cas.cas_enabled:
	411	ui_auth_types.add(LoginType.SSO)
	412
	413	return ui_auth_types
368	414
369	415	def get_enabled_auth_types(self):
370	416	"""Return the enabled user-interactive authentication types

422	468	all the stages in any of the permitted flows.
423	469	"""
424	470
425		authdict = None
426	471	sid = None # type: Optional[str]
427		if clientdict and "auth" in clientdict:
428		authdict = clientdict["auth"]
429		del clientdict["auth"]
430		if "session" in authdict:
431		sid = authdict["session"]
	472	authdict = clientdict.pop("auth", {})
	473	if "session" in authdict:
	474	sid = authdict["session"]
432	475
433	476	# Convert the URI and method to strings.
434	477	uri = request.uri.decode("utf-8")

533	576
534	577	creds = await self.store.get_completed_ui_auth_stages(session.session_id)
535	578	for f in flows:
	579	# If all the required credentials have been supplied, the user has
	580	# successfully completed the UI auth process!
536	581	if len(set(f) - set(creds)) == 0:
537	582	# it's very useful to know what args are stored, but this can
538	583	# include the password in the case of registering, so only log

708	753	device_id: Optional[str],
709	754	valid_until_ms: Optional[int],
710	755	puppets_user_id: Optional[str] = None,
	756	is_appservice_ghost: bool = False,
711	757	) -> str:
712	758	"""
713	759	Creates a new access token for the user with the given user ID.

724	770	we should always have a device ID)
725	771	valid_until_ms: when the token is valid until. None for
726	772	no expiry.
	773	is_appservice_ghost: Whether the user is an application ghost user
727	774	Returns:
728	775	The access token for the user's session.
729	776	Raises:

744	791	"Logging in user %s on device %s%s", user_id, device_id, fmt_expiry
745	792	)
746	793
747		await self.auth.check_auth_blocking(user_id)
	794	if (
	795	not is_appservice_ghost
	796	or self.hs.config.appservice.track_appservice_user_ips
	797	):
	798	await self.auth.check_auth_blocking(user_id)
748	799
749	800	access_token = self.macaroon_gen.generate_access_token(user_id)
750	801	await self.store.add_access_token_to_user(

830	881
831	882	async def validate_login(
832	883	self, login_submission: Dict[str, Any], ratelimit: bool = False,
833		) -> Tuple[str, Optional[Callable[[Dict[str, str]], None]]]:
	884	) -> Tuple[str, Optional[Callable[[Dict[str, str]], Awaitable[None]]]]:
834	885	"""Authenticates the user for the /login API
835	886
836	887	Also used by the user-interactive auth flow to validate auth types which don't

973	1024
974	1025	async def _validate_userid_login(
975	1026	self, username: str, login_submission: Dict[str, Any],
976		) -> Tuple[str, Optional[Callable[[Dict[str, str]], None]]]:
	1027	) -> Tuple[str, Optional[Callable[[Dict[str, str]], Awaitable[None]]]]:
977	1028	"""Helper for validate_login
978	1029
979	1030	Handles login, once we've mapped 3pids onto userids

1028	1079	if result:
1029	1080	return result
1030	1081
1031		if login_type == LoginType.PASSWORD and self.hs.config.password_localdb_enabled:
	1082	if login_type == LoginType.PASSWORD and self._password_localdb_enabled:
1032	1083	known_login_type = True
1033	1084
1034	1085	# we've already checked that there is a (valid) password field

1051	1102
1052	1103	async def check_password_provider_3pid(
1053	1104	self, medium: str, address: str, password: str
1054		) -> Tuple[Optional[str], Optional[Callable[[Dict[str, str]], None]]]:
	1105	) -> Tuple[Optional[str], Optional[Callable[[Dict[str, str]], Awaitable[None]]]]:
1055	1106	"""Check if a password provider is able to validate a thirdparty login
1056	1107
1057	1108	Args:

1302	1353	)
1303	1354
1304	1355	async def complete_sso_ui_auth(
1305		self, registered_user_id: str, session_id: str, request: SynapseRequest,
	1356	self, registered_user_id: str, session_id: str, request: Request,
1306	1357	):
1307	1358	"""Having figured out a mxid for this user, complete the HTTP request
1308	1359
1309	1360	Args:
1310	1361	registered_user_id: The registered user ID to complete SSO login for.
	1362	session_id: The ID of the user-interactive auth session.
1311	1363	request: The request to complete.
1312		client_redirect_url: The URL to which to redirect the user at the end of the
1313		process.
1314	1364	"""
1315	1365	# Mark the stage of the authentication as successful.
1316	1366	# Save the user who authenticated with SSO, this will be used to ensure

1326	1376	async def complete_sso_login(
1327	1377	self,
1328	1378	registered_user_id: str,
1329		request: SynapseRequest,
	1379	request: Request,
1330	1380	client_redirect_url: str,
1331	1381	extra_attributes: Optional[JsonDict] = None,
1332	1382	):

1354	1404	def _complete_sso_login(
1355	1405	self,
1356	1406	registered_user_id: str,
1357		request: SynapseRequest,
	1407	request: Request,
1358	1408	client_redirect_url: str,
1359	1409	extra_attributes: Optional[JsonDict] = None,
1360	1410	):

1608	1658
1609	1659	# This might return an awaitable, if it does block the log out
1610	1660	# until it completes.
1611		result = g(user_id=user_id, device_id=device_id, access_token=access_token,)
1612		if inspect.isawaitable(result):
1613		await result
	1661	await maybe_awaitable(
	1662	g(user_id=user_id, device_id=device_id, access_token=access_token,)
	1663	)

+234

-109

synapse/handlers/cas_handler.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		import urllib
16		from typing import TYPE_CHECKING, Dict, Optional, Tuple
	15	import urllib.parse
	16	from typing import TYPE_CHECKING, Dict, Optional
17	17	from xml.etree import ElementTree as ET
18	18
	19	import attr
	20
19	21	from twisted.web.client import PartialDownloadError
20	22
21		from synapse.api.errors import Codes, LoginError
	23	from synapse.api.errors import HttpResponseException
	24	from synapse.handlers.sso import MappingException, UserAttributes
22	25	from synapse.http.site import SynapseRequest
23	26	from synapse.types import UserID, map_username_to_mxid_localpart
24	27

26	29	from synapse.app.homeserver import HomeServer
27	30
28	31	logger = logging.getLogger(__name__)
	32
	33
	34	class CasError(Exception):
	35	"""Used to catch errors when validating the CAS ticket.
	36	"""
	37
	38	def __init__(self, error, error_description=None):
	39	self.error = error
	40	self.error_description = error_description
	41
	42	def __str__(self):
	43	if self.error_description:
	44	return "{}: {}".format(self.error, self.error_description)
	45	return self.error
	46
	47
	48	@attr.s(slots=True, frozen=True)
	49	class CasResponse:
	50	username = attr.ib(type=str)
	51	attributes = attr.ib(type=Dict[str, Optional[str]])
29	52
30	53
31	54	class CasHandler:

39	62	def __init__(self, hs: "HomeServer"):
40	63	self.hs = hs
41	64	self._hostname = hs.hostname
	65	self._store = hs.get_datastore()
42	66	self._auth_handler = hs.get_auth_handler()
43	67	self._registration_handler = hs.get_registration_handler()
44	68

48	72	self._cas_required_attributes = hs.config.cas_required_attributes
49	73
50	74	self._http_client = hs.get_proxied_http_client()
	75
	76	# identifier for the external_ids table
	77	self._auth_provider_id = "cas"
	78
	79	self._sso_handler = hs.get_sso_handler()
51	80
52	81	def _build_service_param(self, args: Dict[str, str]) -> str:
53	82	"""

68	97
69	98	async def _validate_ticket(
70	99	self, ticket: str, service_args: Dict[str, str]
71		) -> Tuple[str, Optional[str]]:
72		"""
73		Validate a CAS ticket with the server, parse the response, and return the user and display name.
	100	) -> CasResponse:
	101	"""
	102	Validate a CAS ticket with the server, and return the parsed the response.
74	103
75	104	Args:
76	105	ticket: The CAS ticket from the client.
77	106	service_args: Additional arguments to include in the service URL.
78	107	Should be the same as those passed to `get_redirect_url`.
	108
	109	Raises:
	110	CasError: If there's an error parsing the CAS response.
	111
	112	Returns:
	113	The parsed CAS response.
79	114	"""
80	115	uri = self._cas_server_url + "/proxyValidate"
81	116	args = {

88	123	# Twisted raises this error if the connection is closed,
89	124	# even if that's being used old-http style to signal end-of-data
90	125	body = pde.response
91
92		user, attributes = self._parse_cas_response(body)
93		displayname = attributes.pop(self._cas_displayname_attribute, None)
94
95		for required_attribute, required_value in self._cas_required_attributes.items():
96		# If required attribute was not in CAS Response - Forbidden
97		if required_attribute not in attributes:
98		raise LoginError(401, "Unauthorized", errcode=Codes.UNAUTHORIZED)
99
100		# Also need to check value
101		if required_value is not None:
102		actual_value = attributes[required_attribute]
103		# If required attribute value does not match expected - Forbidden
104		if required_value != actual_value:
105		raise LoginError(401, "Unauthorized", errcode=Codes.UNAUTHORIZED)
106
107		return user, displayname
108
109		def _parse_cas_response(
110		self, cas_response_body: bytes
111		) -> Tuple[str, Dict[str, Optional[str]]]:
	126	except HttpResponseException as e:
	127	description = (
	128	(
	129	'Authorization server responded with a "{status}" error '
	130	"while exchanging the authorization code."
	131	).format(status=e.code),
	132	)
	133	raise CasError("server_error", description) from e
	134
	135	return self._parse_cas_response(body)
	136
	137	def _parse_cas_response(self, cas_response_body: bytes) -> CasResponse:
112	138	"""
113	139	Retrieve the user and other parameters from the CAS response.
114	140
115	141	Args:
116	142	cas_response_body: The response from the CAS query.
117	143
	144	Raises:
	145	CasError: If there's an error parsing the CAS response.
	146
118	147	Returns:
119		A tuple of the user and a mapping of other attributes.
120		"""
	148	The parsed CAS response.
	149	"""
	150
	151	# Ensure the response is valid.
	152	root = ET.fromstring(cas_response_body)
	153	if not root.tag.endswith("serviceResponse"):
	154	raise CasError(
	155	"missing_service_response",
	156	"root of CAS response is not serviceResponse",
	157	)
	158
	159	success = root[0].tag.endswith("authenticationSuccess")
	160	if not success:
	161	raise CasError("unsucessful_response", "Unsuccessful CAS response")
	162
	163	# Iterate through the nodes and pull out the user and any extra attributes.
121	164	user = None
122	165	attributes = {}
123		try:
124		root = ET.fromstring(cas_response_body)
125		if not root.tag.endswith("serviceResponse"):
126		raise Exception("root of CAS response is not serviceResponse")
127		success = root[0].tag.endswith("authenticationSuccess")
128		for child in root[0]:
129		if child.tag.endswith("user"):
130		user = child.text
131		if child.tag.endswith("attributes"):
132		for attribute in child:
133		# ElementTree library expands the namespace in
134		# attribute tags to the full URL of the namespace.
135		# We don't care about namespace here and it will always
136		# be encased in curly braces, so we remove them.
137		tag = attribute.tag
138		if "}" in tag:
139		tag = tag.split("}")[1]
140		attributes[tag] = attribute.text
141		if user is None:
142		raise Exception("CAS response does not contain user")
143		except Exception:
144		logger.exception("Error parsing CAS response")
145		raise LoginError(401, "Invalid CAS response", errcode=Codes.UNAUTHORIZED)
146		if not success:
147		raise LoginError(
148		401, "Unsuccessful CAS response", errcode=Codes.UNAUTHORIZED
149		)
150		return user, attributes
	166	for child in root[0]:
	167	if child.tag.endswith("user"):
	168	user = child.text
	169	if child.tag.endswith("attributes"):
	170	for attribute in child:
	171	# ElementTree library expands the namespace in
	172	# attribute tags to the full URL of the namespace.
	173	# We don't care about namespace here and it will always
	174	# be encased in curly braces, so we remove them.
	175	tag = attribute.tag
	176	if "}" in tag:
	177	tag = tag.split("}")[1]
	178	attributes[tag] = attribute.text
	179
	180	# Ensure a user was found.
	181	if user is None:
	182	raise CasError("no_user", "CAS response does not contain user")
	183
	184	return CasResponse(user, attributes)
151	185
152	186	def get_redirect_url(self, service_args: Dict[str, str]) -> str:
153	187	"""

200	234	args["redirectUrl"] = client_redirect_url
201	235	if session:
202	236	args["session"] = session
203		username, user_display_name = await self._validate_ticket(ticket, args)
204
205		# Pull out the user-agent and IP from the request.
206		user_agent = request.get_user_agent("")
207		ip_address = self.hs.get_ip_from_request(request)
208
209		# Get the matrix ID from the CAS username.
210		user_id = await self._map_cas_user_to_matrix_user(
211		username, user_display_name, user_agent, ip_address
	237
	238	try:
	239	cas_response = await self._validate_ticket(ticket, args)
	240	except CasError as e:
	241	logger.exception("Could not validate ticket")
	242	self._sso_handler.render_error(request, e.error, e.error_description, 401)
	243	return
	244
	245	await self._handle_cas_response(
	246	request, cas_response, client_redirect_url, session
212	247	)
213	248
	249	async def _handle_cas_response(
	250	self,
	251	request: SynapseRequest,
	252	cas_response: CasResponse,
	253	client_redirect_url: Optional[str],
	254	session: Optional[str],
	255	) -> None:
	256	"""Handle a CAS response to a ticket request.
	257
	258	Assumes that the response has been validated. Maps the user onto an MXID,
	259	registering them if necessary, and returns a response to the browser.
	260
	261	Args:
	262	request: the incoming request from the browser. We'll respond to it with an
	263	HTML page or a redirect
	264
	265	cas_response: The parsed CAS response.
	266
	267	client_redirect_url: the redirectUrl parameter from the `/cas/ticket` HTTP request, if given.
	268	This should be the same as the redirectUrl from the original `/login/sso/redirect` request.
	269
	270	session: The session parameter from the `/cas/ticket` HTTP request, if given.
	271	This should be the UI Auth session id.
	272	"""
	273
	274	# first check if we're doing a UIA
214	275	if session:
215		await self._auth_handler.complete_sso_ui_auth(
216		user_id, session, request,
217		)
218		else:
219		# If this not a UI auth request than there must be a redirect URL.
220		assert client_redirect_url
221
222		await self._auth_handler.complete_sso_login(
223		user_id, request, client_redirect_url
224		)
225
226		async def _map_cas_user_to_matrix_user(
	276	return await self._sso_handler.complete_sso_ui_auth_request(
	277	self._auth_provider_id, cas_response.username, session, request,
	278	)
	279
	280	# otherwise, we're handling a login request.
	281
	282	# Ensure that the attributes of the logged in user meet the required
	283	# attributes.
	284	for required_attribute, required_value in self._cas_required_attributes.items():
	285	# If required attribute was not in CAS Response - Forbidden
	286	if required_attribute not in cas_response.attributes:
	287	self._sso_handler.render_error(
	288	request,
	289	"unauthorised",
	290	"You are not authorised to log in here.",
	291	401,
	292	)
	293	return
	294
	295	# Also need to check value
	296	if required_value is not None:
	297	actual_value = cas_response.attributes[required_attribute]
	298	# If required attribute value does not match expected - Forbidden
	299	if required_value != actual_value:
	300	self._sso_handler.render_error(
	301	request,
	302	"unauthorised",
	303	"You are not authorised to log in here.",
	304	401,
	305	)
	306	return
	307
	308	# Call the mapper to register/login the user
	309
	310	# If this not a UI auth request than there must be a redirect URL.
	311	assert client_redirect_url is not None
	312
	313	try:
	314	await self._complete_cas_login(cas_response, request, client_redirect_url)
	315	except MappingException as e:
	316	logger.exception("Could not map user")
	317	self._sso_handler.render_error(request, "mapping_error", str(e))
	318
	319	async def _complete_cas_login(
227	320	self,
228		remote_user_id: str,
229		display_name: Optional[str],
230		user_agent: str,
231		ip_address: str,
232		) -> str:
233		"""
234		Given a CAS username, retrieve the user ID for it and possibly register the user.
235
236		Args:
237		remote_user_id: The username from the CAS response.
238		display_name: The display name from the CAS response.
239		user_agent: The user agent of the client making the request.
240		ip_address: The IP address of the client making the request.
241
242		Returns:
243		The user ID associated with this response.
244		"""
245
246		localpart = map_username_to_mxid_localpart(remote_user_id)
247		user_id = UserID(localpart, self._hostname).to_string()
248		registered_user_id = await self._auth_handler.check_user_exists(user_id)
249
250		# If the user does not exist, register it.
251		if not registered_user_id:
252		registered_user_id = await self._registration_handler.register_user(
253		localpart=localpart,
254		default_display_name=display_name,
255		user_agent_ips=[(user_agent, ip_address)],
256		)
257
258		return registered_user_id
	321	cas_response: CasResponse,
	322	request: SynapseRequest,
	323	client_redirect_url: str,
	324	) -> None:
	325	"""
	326	Given a CAS response, complete the login flow
	327
	328	Retrieves the remote user ID, registers the user if necessary, and serves
	329	a redirect back to the client with a login-token.
	330
	331	Args:
	332	cas_response: The parsed CAS response.
	333	request: The request to respond to
	334	client_redirect_url: The redirect URL passed in by the client.
	335
	336	Raises:
	337	MappingException if there was a problem mapping the response to a user.
	338	RedirectException: some mapping providers may raise this if they need
	339	to redirect to an interstitial page.
	340	"""
	341	# Note that CAS does not support a mapping provider, so the logic is hard-coded.
	342	localpart = map_username_to_mxid_localpart(cas_response.username)
	343
	344	async def cas_response_to_user_attributes(failures: int) -> UserAttributes:
	345	"""
	346	Map from CAS attributes to user attributes.
	347	"""
	348	# Due to the grandfathering logic matching any previously registered
	349	# mxids it isn't expected for there to be any failures.
	350	if failures:
	351	raise RuntimeError("CAS is not expected to de-duplicate Matrix IDs")
	352
	353	display_name = cas_response.attributes.get(
	354	self._cas_displayname_attribute, None
	355	)
	356
	357	return UserAttributes(localpart=localpart, display_name=display_name)
	358
	359	async def grandfather_existing_users() -> Optional[str]:
	360	# Since CAS did not always use the user_external_ids table, always
	361	# to attempt to map to existing users.
	362	user_id = UserID(localpart, self._hostname).to_string()
	363
	364	logger.debug(
	365	"Looking for existing account based on mapped %s", user_id,
	366	)
	367
	368	users = await self._store.get_users_by_id_case_insensitive(user_id)
	369	if users:
	370	registered_user_id = list(users.keys())[0]
	371	logger.info("Grandfathering mapping to %s", registered_user_id)
	372	return registered_user_id
	373
	374	return None
	375
	376	await self._sso_handler.complete_sso_login_request(
	377	self._auth_provider_id,
	378	cas_response.username,
	379	request,
	380	client_redirect_url,
	381	cas_response_to_user_attributes,
	382	grandfather_existing_users,
	383	)

+4

-2

synapse/handlers/directory.py less more

132	132	403, "You must be in the room to create an alias for it"
133	133	)
134	134
135		if not self.spam_checker.user_may_create_room_alias(user_id, room_alias):
	135	if not await self.spam_checker.user_may_create_room_alias(
	136	user_id, room_alias
	137	):
136	138	raise AuthError(403, "This user is not permitted to create this alias")
137	139
138	140	if not self.config.is_alias_creation_allowed(

408	410	"""
409	411	user_id = requester.user.to_string()
410	412
411		if not self.spam_checker.user_may_publish_room(user_id, room_id):
	413	if not await self.spam_checker.user_may_publish_room(user_id, room_id):
412	414	raise AuthError(
413	415	403, "This user is not permitted to publish rooms to the room list"
414	416	)

+2

-2

synapse/handlers/federation.py less more

139	139	self._message_handler = hs.get_message_handler()
140	140	self._server_notices_mxid = hs.config.server_notices_mxid
141	141	self.config = hs.config
142		self.http_client = hs.get_simple_http_client()
	142	self.http_client = hs.get_proxied_blacklisted_http_client()
143	143	self._instance_name = hs.get_instance_name()
144	144	self._replication = hs.get_replication_data_handler()
145	145

1592	1592	if self.hs.config.block_non_admin_invites:
1593	1593	raise SynapseError(403, "This server does not accept room invites")
1594	1594
1595		if not self.spam_checker.user_may_invite(
	1595	if not await self.spam_checker.user_may_invite(
1596	1596	event.sender, event.state_key, event.room_id
1597	1597	):
1598	1598	raise SynapseError(

+1

-1

synapse/handlers/groups_local.py less more

28	28
29	29	async def f(self, group_id, args, *kwargs):
30	30	if not GroupID.is_valid(group_id):
31		raise SynapseError(400, "%s was not legal group ID" % (group_id,))
	31	raise SynapseError(400, "%s is not a legal group ID" % (group_id,))
32	32
33	33	if self.is_mine_id(group_id):
34	34	return await getattr(self.groups_server_handler, func_name)(

+8

-3

synapse/handlers/identity.py less more

45	45	def __init__(self, hs):
46	46	super().__init__(hs)
47	47
	48	# An HTTP client for contacting trusted URLs.
48	49	self.http_client = SimpleHttpClient(hs)
49		# We create a blacklisting instance of SimpleHttpClient for contacting identity
50		# servers specified by clients
	50	# An HTTP client for contacting identity servers specified by clients.
51	51	self.blacklisting_http_client = SimpleHttpClient(
52	52	hs, ip_blacklist=hs.config.federation_ip_range_blacklist
53	53	)
54		self.federation_http_client = hs.get_http_client()
	54	self.federation_http_client = hs.get_federation_http_client()
55	55	self.hs = hs
	56
	57	self._web_client_location = hs.config.invite_client_location
56	58
57	59	async def threepid_from_creds(
58	60	self, id_server: str, creds: Dict[str, str]

802	804	"sender_display_name": inviter_display_name,
803	805	"sender_avatar_url": inviter_avatar_url,
804	806	}
	807	# If a custom web client location is available, include it in the request.
	808	if self._web_client_location:
	809	invite_config["org.matrix.web_client_location"] = self._web_client_location
805	810
806	811	# Add the identity service access token to the JSON body and use the v2
807	812	# Identity Service endpoints if id_access_token is present

+1

-3

synapse/handlers/initial_sync.py less more

322	322	member_event_id: str,
323	323	is_peeking: bool,
324	324	) -> JsonDict:
325		room_state = await self.state_store.get_state_for_events([member_event_id])
326
327		room_state = room_state[member_event_id]
	325	room_state = await self.state_store.get_state_for_event(member_event_id)
328	326
329	327	limit = pagin_config.limit if pagin_config else None
330	328	if limit is None:

+2

-2

synapse/handlers/message.py less more

743	743	event.sender,
744	744	)
745	745
746		spam_error = self.spam_checker.check_event_for_spam(event)
	746	spam_error = await self.spam_checker.check_event_for_spam(event)
747	747	if spam_error:
748	748	if not isinstance(spam_error, str):
749	749	spam_error = "Spam is not permitted here"

1260	1260	event, context = await self.create_event(
1261	1261	requester,
1262	1262	{
1263		"type": "org.matrix.dummy_event",
	1263	"type": EventTypes.Dummy,
1264	1264	"content": {},
1265	1265	"room_id": room_id,
1266	1266	"sender": user_id,

+85

-78

synapse/handlers/oidc_handler.py less more

114	114	self._allow_existing_users = hs.config.oidc_allow_existing_users # type: bool
115	115
116	116	self._http_client = hs.get_proxied_http_client()
117		self._auth_handler = hs.get_auth_handler()
118		self._registration_handler = hs.get_registration_handler()
119	117	self._server_name = hs.config.server_name # type: str
120	118	self._macaroon_secret_key = hs.config.macaroon_secret_key
121	119

673	671	self._sso_handler.render_error(request, "invalid_token", str(e))
674	672	return
675	673
676		# Pull out the user-agent and IP from the request.
677		user_agent = request.get_user_agent("")
678		ip_address = self.hs.get_ip_from_request(request)
	674	# first check if we're doing a UIA
	675	if ui_auth_session_id:
	676	try:
	677	remote_user_id = self._remote_id_from_userinfo(userinfo)
	678	except Exception as e:
	679	logger.exception("Could not extract remote user id")
	680	self._sso_handler.render_error(request, "mapping_error", str(e))
	681	return
	682
	683	return await self._sso_handler.complete_sso_ui_auth_request(
	684	self._auth_provider_id, remote_user_id, ui_auth_session_id, request
	685	)
	686
	687	# otherwise, it's a login
679	688
680	689	# Call the mapper to register/login the user
681	690	try:
682		user_id = await self._map_userinfo_to_user(
683		userinfo, token, user_agent, ip_address
	691	await self._complete_oidc_login(
	692	userinfo, token, request, client_redirect_url
684	693	)
685	694	except MappingException as e:
686	695	logger.exception("Could not map user")
687	696	self._sso_handler.render_error(request, "mapping_error", str(e))
688		return
689
690		# Mapping providers might not have get_extra_attributes: only call this
691		# method if it exists.
692		extra_attributes = None
693		get_extra_attributes = getattr(
694		self._user_mapping_provider, "get_extra_attributes", None
695		)
696		if get_extra_attributes:
697		extra_attributes = await get_extra_attributes(userinfo, token)
698
699		# and finally complete the login
700		if ui_auth_session_id:
701		await self._auth_handler.complete_sso_ui_auth(
702		user_id, ui_auth_session_id, request
703		)
704		else:
705		await self._auth_handler.complete_sso_login(
706		user_id, request, client_redirect_url, extra_attributes
707		)
708	697
709	698	def _generate_oidc_session_token(
710	699	self,

827	816	now = self.clock.time_msec()
828	817	return now < expiry
829	818
830		async def _map_userinfo_to_user(
831		self, userinfo: UserInfo, token: Token, user_agent: str, ip_address: str
832		) -> str:
833		"""Maps a UserInfo object to a mxid.
	819	async def _complete_oidc_login(
	820	self,
	821	userinfo: UserInfo,
	822	token: Token,
	823	request: SynapseRequest,
	824	client_redirect_url: str,
	825	) -> None:
	826	"""Given a UserInfo response, complete the login flow
834	827
835	828	UserInfo should have a claim that uniquely identifies users. This claim
836	829	is usually `sub`, but can be configured with `oidc_config.subject_claim`.

842	835	If a user already exists with the mxid we've mapped and allow_existing_users
843	836	is disabled, raise an exception.
844	837
	838	Otherwise, render a redirect back to the client_redirect_url with a loginToken.
	839
845	840	Args:
846	841	userinfo: an object representing the user
847	842	token: a dict with the tokens obtained from the provider
848		user_agent: The user agent of the client making the request.
849		ip_address: The IP address of the client making the request.
	843	request: The request to respond to
	844	client_redirect_url: The redirect URL passed in by the client.
850	845
851	846	Raises:
852	847	MappingException: if there was an error while mapping some properties
853
854		Returns:
855		The mxid of the user
856	848	"""
857	849	try:
858		remote_user_id = self._user_mapping_provider.get_remote_user_id(userinfo)
	850	remote_user_id = self._remote_id_from_userinfo(userinfo)
859	851	except Exception as e:
860	852	raise MappingException(
861	853	"Failed to extract subject from OIDC response: %s" % (e,)
862	854	)
863		# Some OIDC providers use integer IDs, but Synapse expects external IDs
864		# to be strings.
865		remote_user_id = str(remote_user_id)
866	855
867	856	# Older mapping providers don't accept the `failures` argument, so we
868	857	# try and detect support.

923	912
924	913	return None
925	914
926		return await self._sso_handler.get_mxid_from_sso(
	915	# Mapping providers might not have get_extra_attributes: only call this
	916	# method if it exists.
	917	extra_attributes = None
	918	get_extra_attributes = getattr(
	919	self._user_mapping_provider, "get_extra_attributes", None
	920	)
	921	if get_extra_attributes:
	922	extra_attributes = await get_extra_attributes(userinfo, token)
	923
	924	await self._sso_handler.complete_sso_login_request(
927	925	self._auth_provider_id,
928	926	remote_user_id,
929		user_agent,
930		ip_address,
	927	request,
	928	client_redirect_url,
931	929	oidc_response_to_user_attributes,
932	930	grandfather_existing_users,
933		)
	931	extra_attributes,
	932	)
	933
	934	def _remote_id_from_userinfo(self, userinfo: UserInfo) -> str:
	935	"""Extract the unique remote id from an OIDC UserInfo block
	936
	937	Args:
	938	userinfo: An object representing the user given by the OIDC provider
	939	Returns:
	940	remote user id
	941	"""
	942	remote_user_id = self._user_mapping_provider.get_remote_user_id(userinfo)
	943	# Some OIDC providers use integer IDs, but Synapse expects external IDs
	944	# to be strings.
	945	return str(remote_user_id)
934	946
935	947
936	948	UserAttributeDict = TypedDict(
937		"UserAttributeDict", {"localpart": str, "display_name": Optional[str]}
	949	"UserAttributeDict", {"localpart": Optional[str], "display_name": Optional[str]}
938	950	)
939	951	C = TypeVar("C")
940	952

1015	1027
1016	1028	@attr.s
1017	1029	class JinjaOidcMappingConfig:
1018		subject_claim = attr.ib() # type: str
1019		localpart_template = attr.ib() # type: Template
1020		display_name_template = attr.ib() # type: Optional[Template]
1021		extra_attributes = attr.ib() # type: Dict[str, Template]
	1030	subject_claim = attr.ib(type=str)
	1031	localpart_template = attr.ib(type=Optional[Template])
	1032	display_name_template = attr.ib(type=Optional[Template])
	1033	extra_attributes = attr.ib(type=Dict[str, Template])
1022	1034
1023	1035
1024	1036	class JinjaOidcMappingProvider(OidcMappingProvider[JinjaOidcMappingConfig]):

1034	1046	def parse_config(config: dict) -> JinjaOidcMappingConfig:
1035	1047	subject_claim = config.get("subject_claim", "sub")
1036	1048
1037		if "localpart_template" not in config:
1038		raise ConfigError(
1039		"missing key: oidc_config.user_mapping_provider.config.localpart_template"
1040		)
1041
1042		try:
1043		localpart_template = env.from_string(config["localpart_template"])
1044		except Exception as e:
1045		raise ConfigError(
1046		"invalid jinja template for oidc_config.user_mapping_provider.config.localpart_template: %r"
1047		% (e,)
1048		)
	1049	localpart_template = None # type: Optional[Template]
	1050	if "localpart_template" in config:
	1051	try:
	1052	localpart_template = env.from_string(config["localpart_template"])
	1053	except Exception as e:
	1054	raise ConfigError(
	1055	"invalid jinja template", path=["localpart_template"]
	1056	) from e
1049	1057
1050	1058	display_name_template = None # type: Optional[Template]
1051	1059	if "display_name_template" in config:

1053	1061	display_name_template = env.from_string(config["display_name_template"])
1054	1062	except Exception as e:
1055	1063	raise ConfigError(
1056		"invalid jinja template for oidc_config.user_mapping_provider.config.display_name_template: %r"
1057		% (e,)
1058		)
	1064	"invalid jinja template", path=["display_name_template"]
	1065	) from e
1059	1066
1060	1067	extra_attributes = {} # type Dict[str, Template]
1061	1068	if "extra_attributes" in config:
1062	1069	extra_attributes_config = config.get("extra_attributes") or {}
1063	1070	if not isinstance(extra_attributes_config, dict):
1064		raise ConfigError(
1065		"oidc_config.user_mapping_provider.config.extra_attributes must be a dict"
1066		)
	1071	raise ConfigError("must be a dict", path=["extra_attributes"])
1067	1072
1068	1073	for key, value in extra_attributes_config.items():
1069	1074	try:
1070	1075	extra_attributes[key] = env.from_string(value)
1071	1076	except Exception as e:
1072	1077	raise ConfigError(
1073		"invalid jinja template for oidc_config.user_mapping_provider.config.extra_attributes.%s: %r"
1074		% (key, e)
1075		)
	1078	"invalid jinja template", path=["extra_attributes", key]
	1079	) from e
1076	1080
1077	1081	return JinjaOidcMappingConfig(
1078	1082	subject_claim=subject_claim,

1087	1091	async def map_user_attributes(
1088	1092	self, userinfo: UserInfo, token: Token, failures: int
1089	1093	) -> UserAttributeDict:
1090		localpart = self._config.localpart_template.render(user=userinfo).strip()
1091
1092		# Ensure only valid characters are included in the MXID.
1093		localpart = map_username_to_mxid_localpart(localpart)
1094
1095		# Append suffix integer if last call to this function failed to produce
1096		# a usable mxid.
1097		localpart += str(failures) if failures else ""
	1094	localpart = None
	1095
	1096	if self._config.localpart_template:
	1097	localpart = self._config.localpart_template.render(user=userinfo).strip()
	1098
	1099	# Ensure only valid characters are included in the MXID.
	1100	localpart = map_username_to_mxid_localpart(localpart)
	1101
	1102	# Append suffix integer if last call to this function failed to produce
	1103	# a usable mxid.
	1104	localpart += str(failures) if failures else ""
1098	1105
1099	1106	display_name = None # type: Optional[str]
1100	1107	if self._config.display_name_template is not None:

+21

-16

synapse/handlers/receipts.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import List, Tuple
	15	from typing import TYPE_CHECKING, List, Optional, Tuple
16	16
17	17	from synapse.appservice import ApplicationService
18	18	from synapse.handlers._base import BaseHandler
19	19	from synapse.types import JsonDict, ReadReceipt, get_domain_from_id
20		from synapse.util.async_helpers import maybe_awaitable
	20
	21	if TYPE_CHECKING:
	22	from synapse.app.homeserver import HomeServer
21	23
22	24	logger = logging.getLogger(__name__)
23	25
24	26
25	27	class ReceiptsHandler(BaseHandler):
26		def __init__(self, hs):
	28	def __init__(self, hs: "HomeServer"):
27	29	super().__init__(hs)
28	30
29	31	self.server_name = hs.config.server_name

36	38	self.clock = self.hs.get_clock()
37	39	self.state = hs.get_state_handler()
38	40
39		async def _received_remote_receipt(self, origin, content):
	41	async def _received_remote_receipt(self, origin: str, content: JsonDict) -> None:
40	42	"""Called when we receive an EDU of type m.receipt from a remote HS.
41	43	"""
42	44	receipts = []

63	65
64	66	await self._handle_new_receipts(receipts)
65	67
66		async def _handle_new_receipts(self, receipts):
	68	async def _handle_new_receipts(self, receipts: List[ReadReceipt]) -> bool:
67	69	"""Takes a list of receipts, stores them and informs the notifier.
68	70	"""
69		min_batch_id = None
70		max_batch_id = None
	71	min_batch_id = None # type: Optional[int]
	72	max_batch_id = None # type: Optional[int]
71	73
72	74	for receipt in receipts:
73	75	res = await self.store.insert_receipt(

89	91	if max_batch_id is None or max_persisted_id > max_batch_id:
90	92	max_batch_id = max_persisted_id
91	93
92		if min_batch_id is None:
	94	# Either both of these should be None or neither.
	95	if min_batch_id is None or max_batch_id is None:
93	96	# no new receipts
94	97	return False
95	98

97	100
98	101	self.notifier.on_new_event("receipt_key", max_batch_id, rooms=affected_room_ids)
99	102	# Note that the min here shouldn't be relied upon to be accurate.
100		await maybe_awaitable(
101		self.hs.get_pusherpool().on_new_receipts(
102		min_batch_id, max_batch_id, affected_room_ids
103		)
	103	await self.hs.get_pusherpool().on_new_receipts(
	104	min_batch_id, max_batch_id, affected_room_ids
104	105	)
105	106
106	107	return True
107	108
108		async def received_client_receipt(self, room_id, receipt_type, user_id, event_id):
	109	async def received_client_receipt(
	110	self, room_id: str, receipt_type: str, user_id: str, event_id: str
	111	) -> None:
109	112	"""Called when a client tells us a local user has read up to the given
110	113	event_id in the room.
111	114	"""

125	128
126	129
127	130	class ReceiptEventSource:
128		def __init__(self, hs):
	131	def __init__(self, hs: "HomeServer"):
129	132	self.store = hs.get_datastore()
130	133
131		async def get_new_events(self, from_key, room_ids, **kwargs):
	134	async def get_new_events(
	135	self, from_key: int, room_ids: List[str], **kwargs
	136	) -> Tuple[List[JsonDict], int]:
132	137	from_key = int(from_key)
133	138	to_key = self.get_current_key()
134	139

173	178
174	179	return (events, to_key)
175	180
176		def get_current_key(self, direction="f"):
	181	def get_current_key(self, direction: str = "f") -> int:
177	182	return self.store.get_max_receipt_stream_id()

+7

-2

synapse/handlers/register.py less more

186	186	"""
187	187	self.check_registration_ratelimit(address)
188	188
189		result = self.spam_checker.check_registration_for_spam(
	189	result = await self.spam_checker.check_registration_for_spam(
190	190	threepid, localpart, user_agent_ips or [],
191	191	)
192	192

629	629	device_id: Optional[str],
630	630	initial_display_name: Optional[str],
631	631	is_guest: bool = False,
	632	is_appservice_ghost: bool = False,
632	633	) -> Tuple[str, str]:
633	634	"""Register a device for a user and generate an access token.
634	635

650	651	device_id=device_id,
651	652	initial_display_name=initial_display_name,
652	653	is_guest=is_guest,
	654	is_appservice_ghost=is_appservice_ghost,
653	655	)
654	656	return r["device_id"], r["access_token"]
655	657

671	673	)
672	674	else:
673	675	access_token = await self._auth_handler.get_access_token_for_user_id(
674		user_id, device_id=registered_device_id, valid_until_ms=valid_until_ms
	676	user_id,
	677	device_id=registered_device_id,
	678	valid_until_ms=valid_until_ms,
	679	is_appservice_ghost=is_appservice_ghost,
675	680	)
676	681
677	682	return (registered_device_id, access_token)

+10

-6

synapse/handlers/room.py less more

26	26
27	27	from synapse.api.constants import (
28	28	EventTypes,
	29	HistoryVisibility,
29	30	JoinRules,
30	31	Membership,
31	32	RoomCreationPreset,

80	81	self._presets_dict = {
81	82	RoomCreationPreset.PRIVATE_CHAT: {
82	83	"join_rules": JoinRules.INVITE,
83		"history_visibility": "shared",
	84	"history_visibility": HistoryVisibility.SHARED,
84	85	"original_invitees_have_ops": False,
85	86	"guest_can_join": True,
86	87	"power_level_content_override": {"invite": 0},
87	88	},
88	89	RoomCreationPreset.TRUSTED_PRIVATE_CHAT: {
89	90	"join_rules": JoinRules.INVITE,
90		"history_visibility": "shared",
	91	"history_visibility": HistoryVisibility.SHARED,
91	92	"original_invitees_have_ops": True,
92	93	"guest_can_join": True,
93	94	"power_level_content_override": {"invite": 0},
94	95	},
95	96	RoomCreationPreset.PUBLIC_CHAT: {
96	97	"join_rules": JoinRules.PUBLIC,
97		"history_visibility": "shared",
	98	"history_visibility": HistoryVisibility.SHARED,
98	99	"original_invitees_have_ops": False,
99	100	"guest_can_join": False,
100	101	"power_level_content_override": {},

357	358	"""
358	359	user_id = requester.user.to_string()
359	360
360		if not self.spam_checker.user_may_create_room(user_id):
	361	if not await self.spam_checker.user_may_create_room(user_id):
361	362	raise SynapseError(403, "You are not permitted to create rooms")
362	363
363	364	creation_content = {

439	440	invite_list=[],
440	441	initial_state=initial_state,
441	442	creation_content=creation_content,
	443	ratelimit=False,
442	444	)
443	445
444	446	# Transfer membership events

607	609	403, "You are not permitted to create rooms", Codes.FORBIDDEN
608	610	)
609	611
610		if not is_requester_admin and not self.spam_checker.user_may_create_room(
	612	if not is_requester_admin and not await self.spam_checker.user_may_create_room(
611	613	user_id
612	614	):
613	615	raise SynapseError(403, "You are not permitted to create rooms")

734	736	room_alias=room_alias,
735	737	power_level_content_override=power_level_content_override,
736	738	creator_join_profile=creator_join_profile,
	739	ratelimit=ratelimit,
737	740	)
738	741
739	742	if "name" in config:

837	840	room_alias: Optional[RoomAlias] = None,
838	841	power_level_content_override: Optional[JsonDict] = None,
839	842	creator_join_profile: Optional[JsonDict] = None,
	843	ratelimit: bool = True,
840	844	) -> int:
841	845	"""Sends the initial events into a new room.
842	846

883	887	creator.user,
884	888	room_id,
885	889	"join",
886		ratelimit=False,
	890	ratelimit=ratelimit,
887	891	content=creator_join_profile,
888	892	)
889	893

+55

-46

synapse/handlers/room_list.py less more

14	14
15	15	import logging
16	16	from collections import namedtuple
17		from typing import Any, Dict, Optional
	17	from typing import TYPE_CHECKING, Optional, Tuple
18	18
19	19	import msgpack
20	20	from unpaddedbase64 import decode_base64, encode_base64
21	21
22		from synapse.api.constants import EventTypes, JoinRules
	22	from synapse.api.constants import EventTypes, HistoryVisibility, JoinRules
23	23	from synapse.api.errors import Codes, HttpResponseException
24		from synapse.types import ThirdPartyInstanceID
	24	from synapse.types import JsonDict, ThirdPartyInstanceID
25	25	from synapse.util.caches.descriptors import cached
26	26	from synapse.util.caches.response_cache import ResponseCache
27	27
28	28	from ._base import BaseHandler
29	29
	30	if TYPE_CHECKING:
	31	from synapse.app.homeserver import HomeServer
	32
30	33	logger = logging.getLogger(__name__)
31	34
32	35	REMOTE_ROOM_LIST_POLL_INTERVAL = 60 * 1000

36	39
37	40
38	41	class RoomListHandler(BaseHandler):
39		def __init__(self, hs):
	42	def __init__(self, hs: "HomeServer"):
40	43	super().__init__(hs)
41	44	self.enable_room_list_search = hs.config.enable_room_list_search
42		self.response_cache = ResponseCache(hs, "room_list")
	45	self.response_cache = ResponseCache(
	46	hs, "room_list"
	47	) # type: ResponseCache[Tuple[Optional[int], Optional[str], ThirdPartyInstanceID]]
43	48	self.remote_response_cache = ResponseCache(
44	49	hs, "remote_room_list", timeout_ms=30 * 1000
45		)
	50	) # type: ResponseCache[Tuple[str, Optional[int], Optional[str], bool, Optional[str]]]
46	51
47	52	async def get_local_public_room_list(
48	53	self,
49		limit=None,
50		since_token=None,
51		search_filter=None,
52		network_tuple=EMPTY_THIRD_PARTY_ID,
53		from_federation=False,
54		):
	54	limit: Optional[int] = None,
	55	since_token: Optional[str] = None,
	56	search_filter: Optional[dict] = None,
	57	network_tuple: ThirdPartyInstanceID = EMPTY_THIRD_PARTY_ID,
	58	from_federation: bool = False,
	59	) -> JsonDict:
55	60	"""Generate a local public room list.
56	61
57	62	There are multiple different lists: the main one plus one per third
58	63	party network. A client can ask for a specific list or to return all.
59	64
60	65	Args:
61		limit (int\|None)
62		since_token (str\|None)
63		search_filter (dict\|None)
64		network_tuple (ThirdPartyInstanceID): Which public list to use.
	66	limit
	67	since_token
	68	search_filter
	69	network_tuple: Which public list to use.
65	70	This can be (None, None) to indicate the main list, or a particular
66	71	appservice and network id to use an appservice specific one.
67	72	Setting to None returns all public rooms across all lists.
68		from_federation (bool): true iff the request comes from the federation
69		API
	73	from_federation: true iff the request comes from the federation API
70	74	"""
71	75	if not self.enable_room_list_search:
72	76	return {"chunk": [], "total_room_count_estimate": 0}

106	110	self,
107	111	limit: Optional[int] = None,
108	112	since_token: Optional[str] = None,
109		search_filter: Optional[Dict] = None,
	113	search_filter: Optional[dict] = None,
110	114	network_tuple: ThirdPartyInstanceID = EMPTY_THIRD_PARTY_ID,
111	115	from_federation: bool = False,
112		) -> Dict[str, Any]:
	116	) -> JsonDict:
113	117	"""Generate a public room list.
114	118	Args:
115	119	limit: Maximum amount of rooms to return.

130	134	if since_token:
131	135	batch_token = RoomListNextBatch.from_token(since_token)
132	136
133		bounds = (batch_token.last_joined_members, batch_token.last_room_id)
	137	bounds = (
	138	batch_token.last_joined_members,
	139	batch_token.last_room_id,
	140	) # type: Optional[Tuple[int, str]]
134	141	forwards = batch_token.direction_is_forward
	142	has_batch_token = True
135	143	else:
136		batch_token = None
137	144	bounds = None
138	145
139	146	forwards = True
	147	has_batch_token = False
140	148
141	149	# we request one more than wanted to see if there are more pages to come
142	150	probing_limit = limit + 1 if limit is not None else None

158	166	"canonical_alias": room["canonical_alias"],
159	167	"num_joined_members": room["joined_members"],
160	168	"avatar_url": room["avatar"],
161		"world_readable": room["history_visibility"] == "world_readable",
	169	"world_readable": room["history_visibility"]
	170	== HistoryVisibility.WORLD_READABLE,
162	171	"guest_can_join": room["guest_access"] == "can_join",
163	172	}
164	173

167	176
168	177	results = [build_room_entry(r) for r in results]
169	178
170		response = {}
	179	response = {} # type: JsonDict
171	180	num_results = len(results)
172	181	if limit is not None:
173	182	more_to_come = num_results == probing_limit

185	194	initial_entry = results[0]
186	195
187	196	if forwards:
188		if batch_token:
	197	if has_batch_token:
189	198	# If there was a token given then we assume that there
190	199	# must be previous results.
191	200	response["prev_batch"] = RoomListNextBatch(

201	210	direction_is_forward=True,
202	211	).to_token()
203	212	else:
204		if batch_token:
	213	if has_batch_token:
205	214	response["next_batch"] = RoomListNextBatch(
206	215	last_joined_members=final_entry["num_joined_members"],
207	216	last_room_id=final_entry["room_id"],

291	300	return None
292	301
293	302	# Return whether this room is open to federation users or not
294		create_event = current_state.get((EventTypes.Create, ""))
	303	create_event = current_state[EventTypes.Create, ""]
295	304	result["m.federate"] = create_event.content.get("m.federate", True)
296	305
297	306	name_event = current_state.get((EventTypes.Name, ""))

316	325	visibility = None
317	326	if visibility_event:
318	327	visibility = visibility_event.content.get("history_visibility", None)
319		result["world_readable"] = visibility == "world_readable"
	328	result["world_readable"] = visibility == HistoryVisibility.WORLD_READABLE
320	329
321	330	guest_event = current_state.get((EventTypes.GuestAccess, ""))
322	331	guest = None

334	343
335	344	async def get_remote_public_room_list(
336	345	self,
337		server_name,
338		limit=None,
339		since_token=None,
340		search_filter=None,
341		include_all_networks=False,
342		third_party_instance_id=None,
343		):
	346	server_name: str,
	347	limit: Optional[int] = None,
	348	since_token: Optional[str] = None,
	349	search_filter: Optional[dict] = None,
	350	include_all_networks: bool = False,
	351	third_party_instance_id: Optional[str] = None,
	352	) -> JsonDict:
344	353	if not self.enable_room_list_search:
345	354	return {"chunk": [], "total_room_count_estimate": 0}
346	355

397	406
398	407	async def _get_remote_list_cached(
399	408	self,
400		server_name,
401		limit=None,
402		since_token=None,
403		search_filter=None,
404		include_all_networks=False,
405		third_party_instance_id=None,
406		):
	409	server_name: str,
	410	limit: Optional[int] = None,
	411	since_token: Optional[str] = None,
	412	search_filter: Optional[dict] = None,
	413	include_all_networks: bool = False,
	414	third_party_instance_id: Optional[str] = None,
	415	) -> JsonDict:
407	416	repl_layer = self.hs.get_federation_client()
408	417	if search_filter:
409	418	# We can't cache when asking for search

454	463	REVERSE_KEY_DICT = {v: k for k, v in KEY_DICT.items()}
455	464
456	465	@classmethod
457		def from_token(cls, token):
	466	def from_token(cls, token: str) -> "RoomListNextBatch":
458	467	decoded = msgpack.loads(decode_base64(token), raw=False)
459	468	return RoomListNextBatch(
460	469	**{cls.REVERSE_KEY_DICT[key]: val for key, val in decoded.items()}
461	470	)
462	471
463		def to_token(self):
	472	def to_token(self) -> str:
464	473	return encode_base64(
465	474	msgpack.dumps(
466	475	{self.KEY_DICT[key]: val for key, val in self._asdict().items()}
467	476	)
468	477	)
469	478
470		def copy_and_replace(self, **kwds):
	479	def copy_and_replace(self, **kwds) -> "RoomListNextBatch":
471	480	return self._replace(**kwds)
472	481
473	482
474		def _matches_room_entry(room_entry, search_filter):
	483	def _matches_room_entry(room_entry: JsonDict, search_filter: dict) -> bool:
475	484	if search_filter and search_filter.get("generic_search_term", None):
476	485	generic_search_term = search_filter["generic_search_term"].upper()
477	486	if generic_search_term in room_entry.get("name", "").upper():

+14

-11

synapse/handlers/room_member.py less more

202	202
203	203	# Only rate-limit if the user actually joined the room, otherwise we'll end
204	204	# up blocking profile updates.
205		if newly_joined:
	205	if newly_joined and ratelimit:
206	206	time_now_s = self.clock.time()
207	207	(
208	208	allowed,

407	407	)
408	408	block_invite = True
409	409
410		if not self.spam_checker.user_may_invite(
	410	if not await self.spam_checker.user_may_invite(
411	411	requester.user.to_string(), target.to_string(), room_id
412	412	):
413	413	logger.info("Blocking invite due to spam checker")

487	487	raise AuthError(403, "Guest access not allowed")
488	488
489	489	if not is_host_in_room:
490		time_now_s = self.clock.time()
491		(
492		allowed,
493		time_allowed,
494		) = self._join_rate_limiter_remote.can_requester_do_action(requester,)
495
496		if not allowed:
497		raise LimitExceededError(
498		retry_after_ms=int(1000 * (time_allowed - time_now_s))
	490	if ratelimit:
	491	time_now_s = self.clock.time()
	492	(
	493	allowed,
	494	time_allowed,
	495	) = self._join_rate_limiter_remote.can_requester_do_action(
	496	requester,
499	497	)
	498
	499	if not allowed:
	500	raise LimitExceededError(
	501	retry_after_ms=int(1000 * (time_allowed - time_now_s))
	502	)
500	503
501	504	inviter = await self._get_inviter(target.to_string(), room_id)
502	505	if inviter and not self.hs.is_mine(inviter):

+90

-51

synapse/handlers/saml_handler.py less more

33	33	map_username_to_mxid_localpart,
34	34	mxid_localpart_allowed_characters,
35	35	)
36		from synapse.util.async_helpers import Linearizer
37	36	from synapse.util.iterutils import chunk_seq
38	37
39	38	if TYPE_CHECKING:

58	57	super().__init__(hs)
59	58	self._saml_client = Saml2Client(hs.config.saml2_sp_config)
60	59	self._saml_idp_entityid = hs.config.saml2_idp_entityid
61		self._auth_handler = hs.get_auth_handler()
62		self._registration_handler = hs.get_registration_handler()
63	60
64	61	self._saml2_session_lifetime = hs.config.saml2_session_lifetime
65	62	self._grandfathered_mxid_source_attribute = (

79	76
80	77	# a map from saml session id to Saml2SessionData object
81	78	self._outstanding_requests_dict = {} # type: Dict[str, Saml2SessionData]
82
83		# a lock on the mappings
84		self._mapping_lock = Linearizer(name="saml_mapping", clock=self.clock)
85	79
86	80	self._sso_handler = hs.get_sso_handler()
87	81

166	160	return
167	161
168	162	logger.debug("SAML2 response: %s", saml2_auth.origxml)
	163
	164	await self._handle_authn_response(request, saml2_auth, relay_state)
	165
	166	async def _handle_authn_response(
	167	self,
	168	request: SynapseRequest,
	169	saml2_auth: saml2.response.AuthnResponse,
	170	relay_state: str,
	171	) -> None:
	172	"""Handle an AuthnResponse, having parsed it from the request params
	173
	174	Assumes that the signature on the response object has been checked. Maps
	175	the user onto an MXID, registering them if necessary, and returns a response
	176	to the browser.
	177
	178	Args:
	179	request: the incoming request from the browser. We'll respond to it with an
	180	HTML page or a redirect
	181	saml2_auth: the parsed AuthnResponse object
	182	relay_state: the RelayState query param, which encodes the URI to rediret
	183	back to
	184	"""
	185
169	186	for assertion in saml2_auth.assertions:
170	187	# kibana limits the length of a log field, whereas this is all rather
171	188	# useful, so split it up.

182	199	saml2_auth.in_response_to, None
183	200	)
184	201
	202	# first check if we're doing a UIA
	203	if current_session and current_session.ui_auth_session_id:
	204	try:
	205	remote_user_id = self._remote_id_from_saml_response(saml2_auth, None)
	206	except MappingException as e:
	207	logger.exception("Failed to extract remote user id from SAML response")
	208	self._sso_handler.render_error(request, "mapping_error", str(e))
	209	return
	210
	211	return await self._sso_handler.complete_sso_ui_auth_request(
	212	self._auth_provider_id,
	213	remote_user_id,
	214	current_session.ui_auth_session_id,
	215	request,
	216	)
	217
	218	# otherwise, we're handling a login request.
	219
185	220	# Ensure that the attributes of the logged in user meet the required
186	221	# attributes.
187	222	for requirement in self._saml2_attribute_requirements:

191	226	)
192	227	return
193	228
194		# Pull out the user-agent and IP from the request.
195		user_agent = request.get_user_agent("")
196		ip_address = self.hs.get_ip_from_request(request)
197
198	229	# Call the mapper to register/login the user
199	230	try:
200		user_id = await self._map_saml_response_to_user(
201		saml2_auth, relay_state, user_agent, ip_address
202		)
	231	await self._complete_saml_login(saml2_auth, request, relay_state)
203	232	except MappingException as e:
204	233	logger.exception("Could not map user")
205	234	self._sso_handler.render_error(request, "mapping_error", str(e))
206		return
207
208		# Complete the interactive auth session or the login.
209		if current_session and current_session.ui_auth_session_id:
210		await self._auth_handler.complete_sso_ui_auth(
211		user_id, current_session.ui_auth_session_id, request
212		)
213
214		else:
215		await self._auth_handler.complete_sso_login(user_id, request, relay_state)
216
217		async def _map_saml_response_to_user(
	235
	236	async def _complete_saml_login(
218	237	self,
219	238	saml2_auth: saml2.response.AuthnResponse,
	239	request: SynapseRequest,
220	240	client_redirect_url: str,
221		user_agent: str,
222		ip_address: str,
223		) -> str:
224		"""
225		Given a SAML response, retrieve the user ID for it and possibly register the user.
	241	) -> None:
	242	"""
	243	Given a SAML response, complete the login flow
	244
	245	Retrieves the remote user ID, registers the user if necessary, and serves
	246	a redirect back to the client with a login-token.
226	247
227	248	Args:
228	249	saml2_auth: The parsed SAML2 response.
	250	request: The request to respond to
229	251	client_redirect_url: The redirect URL passed in by the client.
230		user_agent: The user agent of the client making the request.
231		ip_address: The IP address of the client making the request.
232
233		Returns:
234		The user ID associated with this response.
235	252
236	253	Raises:
237	254	MappingException if there was a problem mapping the response to a user.
238	255	RedirectException: some mapping providers may raise this if they need
239	256	to redirect to an interstitial page.
240	257	"""
241
242		remote_user_id = self._user_mapping_provider.get_remote_user_id(
	258	remote_user_id = self._remote_id_from_saml_response(
243	259	saml2_auth, client_redirect_url
244	260	)
245
246		if not remote_user_id:
247		raise MappingException(
248		"Failed to extract remote user id from SAML response"
249		)
250	261
251	262	async def saml_response_to_remapped_user_attributes(
252	263	failures: int,

293	304
294	305	return None
295	306
296		with (await self._mapping_lock.queue(self._auth_provider_id)):
297		return await self._sso_handler.get_mxid_from_sso(
298		self._auth_provider_id,
299		remote_user_id,
300		user_agent,
301		ip_address,
302		saml_response_to_remapped_user_attributes,
303		grandfather_existing_users,
304		)
	307	await self._sso_handler.complete_sso_login_request(
	308	self._auth_provider_id,
	309	remote_user_id,
	310	request,
	311	client_redirect_url,
	312	saml_response_to_remapped_user_attributes,
	313	grandfather_existing_users,
	314	)
	315
	316	def _remote_id_from_saml_response(
	317	self,
	318	saml2_auth: saml2.response.AuthnResponse,
	319	client_redirect_url: Optional[str],
	320	) -> str:
	321	"""Extract the unique remote id from a SAML2 AuthnResponse
	322
	323	Args:
	324	saml2_auth: The parsed SAML2 response.
	325	client_redirect_url: The redirect URL passed in by the client.
	326	Returns:
	327	remote user id
	328
	329	Raises:
	330	MappingException if there was an error extracting the user id
	331	"""
	332	# It's not obvious why we need to pass in the redirect URI to the mapping
	333	# provider, but we do :/
	334	remote_user_id = self._user_mapping_provider.get_remote_user_id(
	335	saml2_auth, client_redirect_url
	336	)
	337
	338	if not remote_user_id:
	339	raise MappingException(
	340	"Failed to extract remote user id from SAML response"
	341	)
	342
	343	return remote_user_id
305	344
306	345	def expire_sessions(self):
307	346	expire_before = self.clock.time_msec() - self._saml2_session_lifetime

+372

-45

synapse/handlers/sso.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import TYPE_CHECKING, Awaitable, Callable, List, Optional
	15	from typing import TYPE_CHECKING, Awaitable, Callable, Dict, List, Optional
16	16
17	17	import attr
18
19		from synapse.api.errors import RedirectException
20		from synapse.handlers._base import BaseHandler
	18	from typing_extensions import NoReturn
	19
	20	from twisted.web.http import Request
	21
	22	from synapse.api.errors import RedirectException, SynapseError
21	23	from synapse.http.server import respond_with_html
22		from synapse.types import UserID, contains_invalid_mxid_characters
	24	from synapse.http.site import SynapseRequest
	25	from synapse.types import JsonDict, UserID, contains_invalid_mxid_characters
	26	from synapse.util.async_helpers import Linearizer
	27	from synapse.util.stringutils import random_string
23	28
24	29	if TYPE_CHECKING:
25	30	from synapse.server import HomeServer

36	41
37	42	@attr.s
38	43	class UserAttributes:
39		localpart = attr.ib(type=str)
	44	# the localpart of the mxid that the mapper has assigned to the user.
	45	# if `None`, the mapper has not picked a userid, and the user should be prompted to
	46	# enter one.
	47	localpart = attr.ib(type=Optional[str])
40	48	display_name = attr.ib(type=Optional[str], default=None)
41	49	emails = attr.ib(type=List[str], default=attr.Factory(list))
42	50
43	51
44		class SsoHandler(BaseHandler):
	52	@attr.s(slots=True)
	53	class UsernameMappingSession:
	54	"""Data we track about SSO sessions"""
	55
	56	# A unique identifier for this SSO provider, e.g. "oidc" or "saml".
	57	auth_provider_id = attr.ib(type=str)
	58
	59	# user ID on the IdP server
	60	remote_user_id = attr.ib(type=str)
	61
	62	# attributes returned by the ID mapper
	63	display_name = attr.ib(type=Optional[str])
	64	emails = attr.ib(type=List[str])
	65
	66	# An optional dictionary of extra attributes to be provided to the client in the
	67	# login response.
	68	extra_login_attributes = attr.ib(type=Optional[JsonDict])
	69
	70	# where to redirect the client back to
	71	client_redirect_url = attr.ib(type=str)
	72
	73	# expiry time for the session, in milliseconds
	74	expiry_time_ms = attr.ib(type=int)
	75
	76
	77	# the HTTP cookie used to track the mapping session id
	78	USERNAME_MAPPING_SESSION_COOKIE_NAME = b"username_mapping_session"
	79
	80
	81	class SsoHandler:
45	82	# The number of attempts to ask the mapping provider for when generating an MXID.
46	83	_MAP_USERNAME_RETRIES = 1000
47	84
	85	# the time a UsernameMappingSession remains valid for
	86	_MAPPING_SESSION_VALIDITY_PERIOD_MS = 15 * 60 * 1000
	87
48	88	def __init__(self, hs: "HomeServer"):
49		super().__init__(hs)
	89	self._clock = hs.get_clock()
	90	self._store = hs.get_datastore()
	91	self._server_name = hs.hostname
50	92	self._registration_handler = hs.get_registration_handler()
51	93	self._error_template = hs.config.sso_error_template
	94	self._auth_handler = hs.get_auth_handler()
	95
	96	# a lock on the mappings
	97	self._mapping_lock = Linearizer(name="sso_user_mapping", clock=hs.get_clock())
	98
	99	# a map from session id to session data
	100	self._username_mapping_sessions = {} # type: Dict[str, UsernameMappingSession]
52	101
53	102	def render_error(
54		self, request, error: str, error_description: Optional[str] = None
	103	self,
	104	request: Request,
	105	error: str,
	106	error_description: Optional[str] = None,
	107	code: int = 400,
55	108	) -> None:
56	109	"""Renders the error template and responds with it.
57	110

63	116	We'll respond with an HTML page describing the error.
64	117	error: A technical identifier for this error.
65	118	error_description: A human-readable description of the error.
	119	code: The integer error code (an HTTP response code)
66	120	"""
67	121	html = self._error_template.render(
68	122	error=error, error_description=error_description
69	123	)
70		respond_with_html(request, 400, html)
	124	respond_with_html(request, code, html)
71	125
72	126	async def get_sso_user_by_remote_user_id(
73	127	self, auth_provider_id: str, remote_user_id: str

94	148	)
95	149
96	150	# Check if we already have a mapping for this user.
97		previously_registered_user_id = await self.store.get_user_by_external_id(
	151	previously_registered_user_id = await self._store.get_user_by_external_id(
98	152	auth_provider_id, remote_user_id,
99	153	)
100	154

111	165	# No match.
112	166	return None
113	167
114		async def get_mxid_from_sso(
	168	async def complete_sso_login_request(
115	169	self,
116	170	auth_provider_id: str,
117	171	remote_user_id: str,
118		user_agent: str,
119		ip_address: str,
	172	request: SynapseRequest,
	173	client_redirect_url: str,
120	174	sso_to_matrix_id_mapper: Callable[[int], Awaitable[UserAttributes]],
121		grandfather_existing_users: Optional[Callable[[], Awaitable[Optional[str]]]],
122		) -> str:
	175	grandfather_existing_users: Callable[[], Awaitable[Optional[str]]],
	176	extra_login_attributes: Optional[JsonDict] = None,
	177	) -> None:
123	178	"""
124	179	Given an SSO ID, retrieve the user ID for it and possibly register the user.
125	180

138	193	given user-agent and IP address and the SSO ID is linked to this matrix
139	194	ID for subsequent calls.
140	195
	196	Finally, we generate a redirect to the supplied redirect uri, with a login token
	197
141	198	Args:
142	199	auth_provider_id: A unique identifier for this SSO provider, e.g.
143	200	"oidc" or "saml".
	201
144	202	remote_user_id: The unique identifier from the SSO provider.
145		user_agent: The user agent of the client making the request.
146		ip_address: The IP address of the client making the request.
	203
	204	request: The request to respond to
	205
	206	client_redirect_url: The redirect URL passed in by the client.
	207
147	208	sso_to_matrix_id_mapper: A callable to generate the user attributes.
148	209	The only parameter is an integer which represents the amount of
149	210	times the returned mxid localpart mapping has failed.

155	216	to the user.
156	217	RedirectException to redirect to an additional page (e.g.
157	218	to prompt the user for more information).
	219
158	220	grandfather_existing_users: A callable which can return an previously
159	221	existing matrix ID. The SSO ID is then linked to the returned
160	222	matrix ID.
161	223
162		Returns:
163		The user ID associated with the SSO response.
	224	extra_login_attributes: An optional dictionary of extra
	225	attributes to be provided to the client in the login response.
164	226
165	227	Raises:
166	228	MappingException if there was a problem mapping the response to a user.

168	230	to an additional page. (e.g. to prompt for more information)
169	231
170	232	"""
171		# first of all, check if we already have a mapping for this user
172		previously_registered_user_id = await self.get_sso_user_by_remote_user_id(
173		auth_provider_id, remote_user_id,
174		)
175		if previously_registered_user_id:
176		return previously_registered_user_id
177
178		# Check for grandfathering of users.
179		if grandfather_existing_users:
180		previously_registered_user_id = await grandfather_existing_users()
181		if previously_registered_user_id:
182		# Future logins should also match this user ID.
183		await self.store.record_user_external_id(
184		auth_provider_id, remote_user_id, previously_registered_user_id
	233	# grab a lock while we try to find a mapping for this user. This seems...
	234	# optimistic, especially for implementations that end up redirecting to
	235	# interstitial pages.
	236	with await self._mapping_lock.queue(auth_provider_id):
	237	# first of all, check if we already have a mapping for this user
	238	user_id = await self.get_sso_user_by_remote_user_id(
	239	auth_provider_id, remote_user_id,
	240	)
	241
	242	# Check for grandfathering of users.
	243	if not user_id:
	244	user_id = await grandfather_existing_users()
	245	if user_id:
	246	# Future logins should also match this user ID.
	247	await self._store.record_user_external_id(
	248	auth_provider_id, remote_user_id, user_id
	249	)
	250
	251	# Otherwise, generate a new user.
	252	if not user_id:
	253	attributes = await self._call_attribute_mapper(sso_to_matrix_id_mapper)
	254
	255	if attributes.localpart is None:
	256	# the mapper doesn't return a username. bail out with a redirect to
	257	# the username picker.
	258	await self._redirect_to_username_picker(
	259	auth_provider_id,
	260	remote_user_id,
	261	attributes,
	262	client_redirect_url,
	263	extra_login_attributes,
	264	)
	265
	266	user_id = await self._register_mapped_user(
	267	attributes,
	268	auth_provider_id,
	269	remote_user_id,
	270	request.get_user_agent(""),
	271	request.getClientIP(),
185	272	)
186		return previously_registered_user_id
187
188		# Otherwise, generate a new user.
	273
	274	await self._auth_handler.complete_sso_login(
	275	user_id, request, client_redirect_url, extra_login_attributes
	276	)
	277
	278	async def _call_attribute_mapper(
	279	self, sso_to_matrix_id_mapper: Callable[[int], Awaitable[UserAttributes]],
	280	) -> UserAttributes:
	281	"""Call the attribute mapper function in a loop, until we get a unique userid"""
189	282	for i in range(self._MAP_USERNAME_RETRIES):
190	283	try:
191	284	attributes = await sso_to_matrix_id_mapper(i)

207	300	)
208	301
209	302	if not attributes.localpart:
210		raise MappingException(
211		"Error parsing SSO response: SSO mapping provider plugin "
212		"did not return a localpart value"
213		)
	303	# the mapper has not picked a localpart
	304	return attributes
214	305
215	306	# Check if this mxid already exists
216		user_id = UserID(attributes.localpart, self.server_name).to_string()
217		if not await self.store.get_users_by_id_case_insensitive(user_id):
	307	user_id = UserID(attributes.localpart, self._server_name).to_string()
	308	if not await self._store.get_users_by_id_case_insensitive(user_id):
218	309	# This mxid is free
219	310	break
220	311	else:

223	314	raise MappingException(
224	315	"Unable to generate a Matrix ID from the SSO response"
225	316	)
	317	return attributes
	318
	319	async def _redirect_to_username_picker(
	320	self,
	321	auth_provider_id: str,
	322	remote_user_id: str,
	323	attributes: UserAttributes,
	324	client_redirect_url: str,
	325	extra_login_attributes: Optional[JsonDict],
	326	) -> NoReturn:
	327	"""Creates a UsernameMappingSession and redirects the browser
	328
	329	Called if the user mapping provider doesn't return a localpart for a new user.
	330	Raises a RedirectException which redirects the browser to the username picker.
	331
	332	Args:
	333	auth_provider_id: A unique identifier for this SSO provider, e.g.
	334	"oidc" or "saml".
	335
	336	remote_user_id: The unique identifier from the SSO provider.
	337
	338	attributes: the user attributes returned by the user mapping provider.
	339
	340	client_redirect_url: The redirect URL passed in by the client, which we
	341	will eventually redirect back to.
	342
	343	extra_login_attributes: An optional dictionary of extra
	344	attributes to be provided to the client in the login response.
	345
	346	Raises:
	347	RedirectException
	348	"""
	349	session_id = random_string(16)
	350	now = self._clock.time_msec()
	351	session = UsernameMappingSession(
	352	auth_provider_id=auth_provider_id,
	353	remote_user_id=remote_user_id,
	354	display_name=attributes.display_name,
	355	emails=attributes.emails,
	356	client_redirect_url=client_redirect_url,
	357	expiry_time_ms=now + self._MAPPING_SESSION_VALIDITY_PERIOD_MS,
	358	extra_login_attributes=extra_login_attributes,
	359	)
	360
	361	self._username_mapping_sessions[session_id] = session
	362	logger.info("Recorded registration session id %s", session_id)
	363
	364	# Set the cookie and redirect to the username picker
	365	e = RedirectException(b"/_synapse/client/pick_username")
	366	e.cookies.append(
	367	b"%s=%s; path=/"
	368	% (USERNAME_MAPPING_SESSION_COOKIE_NAME, session_id.encode("ascii"))
	369	)
	370	raise e
	371
	372	async def _register_mapped_user(
	373	self,
	374	attributes: UserAttributes,
	375	auth_provider_id: str,
	376	remote_user_id: str,
	377	user_agent: str,
	378	ip_address: str,
	379	) -> str:
	380	"""Register a new SSO user.
	381
	382	This is called once we have successfully mapped the remote user id onto a local
	383	user id, one way or another.
	384
	385	Args:
	386	attributes: user attributes returned by the user mapping provider,
	387	including a non-empty localpart.
	388
	389	auth_provider_id: A unique identifier for this SSO provider, e.g.
	390	"oidc" or "saml".
	391
	392	remote_user_id: The unique identifier from the SSO provider.
	393
	394	user_agent: The user-agent in the HTTP request (used for potential
	395	shadow-banning.)
	396
	397	ip_address: The IP address of the requester (used for potential
	398	shadow-banning.)
	399
	400	Raises:
	401	a MappingException if the localpart is invalid.
	402
	403	a SynapseError with code 400 and errcode Codes.USER_IN_USE if the localpart
	404	is already taken.
	405	"""
226	406
227	407	# Since the localpart is provided via a potentially untrusted module,
228	408	# ensure the MXID is valid before registering.
229		if contains_invalid_mxid_characters(attributes.localpart):
	409	if not attributes.localpart or contains_invalid_mxid_characters(
	410	attributes.localpart
	411	):
230	412	raise MappingException("localpart is invalid: %s" % (attributes.localpart,))
231	413
232	414	logger.debug("Mapped SSO user to local part %s", attributes.localpart)

237	419	user_agent_ips=[(user_agent, ip_address)],
238	420	)
239	421
240		await self.store.record_user_external_id(
	422	await self._store.record_user_external_id(
241	423	auth_provider_id, remote_user_id, registered_user_id
242	424	)
243	425	return registered_user_id
	426
	427	async def complete_sso_ui_auth_request(
	428	self,
	429	auth_provider_id: str,
	430	remote_user_id: str,
	431	ui_auth_session_id: str,
	432	request: Request,
	433	) -> None:
	434	"""
	435	Given an SSO ID, retrieve the user ID for it and complete UIA.
	436
	437	Note that this requires that the user is mapped in the "user_external_ids"
	438	table. This will be the case if they have ever logged in via SAML or OIDC in
	439	recentish synapse versions, but may not be for older users.
	440
	441	Args:
	442	auth_provider_id: A unique identifier for this SSO provider, e.g.
	443	"oidc" or "saml".
	444	remote_user_id: The unique identifier from the SSO provider.
	445	ui_auth_session_id: The ID of the user-interactive auth session.
	446	request: The request to complete.
	447	"""
	448
	449	user_id = await self.get_sso_user_by_remote_user_id(
	450	auth_provider_id, remote_user_id,
	451	)
	452
	453	if not user_id:
	454	logger.warning(
	455	"Remote user %s/%s has not previously logged in here: UIA will fail",
	456	auth_provider_id,
	457	remote_user_id,
	458	)
	459	# Let the UIA flow handle this the same as if they presented creds for a
	460	# different user.
	461	user_id = ""
	462
	463	await self._auth_handler.complete_sso_ui_auth(
	464	user_id, ui_auth_session_id, request
	465	)
	466
	467	async def check_username_availability(
	468	self, localpart: str, session_id: str,
	469	) -> bool:
	470	"""Handle an "is username available" callback check
	471
	472	Args:
	473	localpart: desired localpart
	474	session_id: the session id for the username picker
	475	Returns:
	476	True if the username is available
	477	Raises:
	478	SynapseError if the localpart is invalid or the session is unknown
	479	"""
	480
	481	# make sure that there is a valid mapping session, to stop people dictionary-
	482	# scanning for accounts
	483
	484	self._expire_old_sessions()
	485	session = self._username_mapping_sessions.get(session_id)
	486	if not session:
	487	logger.info("Couldn't find session id %s", session_id)
	488	raise SynapseError(400, "unknown session")
	489
	490	logger.info(
	491	"[session %s] Checking for availability of username %s",
	492	session_id,
	493	localpart,
	494	)
	495
	496	if contains_invalid_mxid_characters(localpart):
	497	raise SynapseError(400, "localpart is invalid: %s" % (localpart,))
	498	user_id = UserID(localpart, self._server_name).to_string()
	499	user_infos = await self._store.get_users_by_id_case_insensitive(user_id)
	500
	501	logger.info("[session %s] users: %s", session_id, user_infos)
	502	return not user_infos
	503
	504	async def handle_submit_username_request(
	505	self, request: SynapseRequest, localpart: str, session_id: str
	506	) -> None:
	507	"""Handle a request to the username-picker 'submit' endpoint
	508
	509	Will serve an HTTP response to the request.
	510
	511	Args:
	512	request: HTTP request
	513	localpart: localpart requested by the user
	514	session_id: ID of the username mapping session, extracted from a cookie
	515	"""
	516	self._expire_old_sessions()
	517	session = self._username_mapping_sessions.get(session_id)
	518	if not session:
	519	logger.info("Couldn't find session id %s", session_id)
	520	raise SynapseError(400, "unknown session")
	521
	522	logger.info("[session %s] Registering localpart %s", session_id, localpart)
	523
	524	attributes = UserAttributes(
	525	localpart=localpart,
	526	display_name=session.display_name,
	527	emails=session.emails,
	528	)
	529
	530	# the following will raise a 400 error if the username has been taken in the
	531	# meantime.
	532	user_id = await self._register_mapped_user(
	533	attributes,
	534	session.auth_provider_id,
	535	session.remote_user_id,
	536	request.get_user_agent(""),
	537	request.getClientIP(),
	538	)
	539
	540	logger.info("[session %s] Registered userid %s", session_id, user_id)
	541
	542	# delete the mapping session and the cookie
	543	del self._username_mapping_sessions[session_id]
	544
	545	# delete the cookie
	546	request.addCookie(
	547	USERNAME_MAPPING_SESSION_COOKIE_NAME,
	548	b"",
	549	expires=b"Thu, 01 Jan 1970 00:00:00 GMT",
	550	path=b"/",
	551	)
	552
	553	await self._auth_handler.complete_sso_login(
	554	user_id,
	555	request,
	556	session.client_redirect_url,
	557	session.extra_login_attributes,
	558	)
	559
	560	def _expire_old_sessions(self):
	561	to_expire = []
	562	now = int(self._clock.time_msec())
	563
	564	for session_id, session in self._username_mapping_sessions.items():
	565	if session.expiry_time_ms <= now:
	566	to_expire.append(session_id)
	567
	568	for session_id in to_expire:
	569	logger.info("Expiring mapping session %s", session_id)
	570	del self._username_mapping_sessions[session_id]

+1

-1

synapse/handlers/sync.py less more

553	553	event.event_id, state_filter=state_filter
554	554	)
555	555	if event.is_state():
556		state_ids = state_ids.copy()
	556	state_ids = dict(state_ids)
557	557	state_ids[(event.type, event.state_key)] = event.event_id
558	558	return state_ids
559	559

+55

-30

synapse/handlers/user_directory.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import TYPE_CHECKING, Any, Dict, List, Optional
16	17
17	18	import synapse.metrics
18		from synapse.api.constants import EventTypes, JoinRules, Membership
	19	from synapse.api.constants import EventTypes, HistoryVisibility, JoinRules, Membership
19	20	from synapse.handlers.state_deltas import StateDeltasHandler
20	21	from synapse.metrics.background_process_metrics import run_as_background_process
21	22	from synapse.storage.roommember import ProfileInfo
	23	from synapse.types import JsonDict
22	24	from synapse.util.metrics import Measure
	25
	26	if TYPE_CHECKING:
	27	from synapse.app.homeserver import HomeServer
23	28
24	29	logger = logging.getLogger(__name__)
25	30

35	40	be in the directory or not when necessary.
36	41	"""
37	42
38		def __init__(self, hs):
	43	def __init__(self, hs: "HomeServer"):
39	44	super().__init__(hs)
40	45
41	46	self.store = hs.get_datastore()

48	53	self.search_all_users = hs.config.user_directory_search_all_users
49	54	self.spam_checker = hs.get_spam_checker()
50	55	# The current position in the current_state_delta stream
51		self.pos = None
	56	self.pos = None # type: Optional[int]
52	57
53	58	# Guard to ensure we only process deltas one at a time
54	59	self._is_processing = False

60	65	# we start populating the user directory
61	66	self.clock.call_later(0, self.notify_new_event)
62	67
63		async def search_users(self, user_id, search_term, limit):
	68	async def search_users(
	69	self, user_id: str, search_term: str, limit: int
	70	) -> JsonDict:
64	71	"""Searches for users in directory
65	72
66	73	Returns:

80	87	results = await self.store.search_user_dir(user_id, search_term, limit)
81	88
82	89	# Remove any spammy users from the results.
83		results["results"] = [
84		user
85		for user in results["results"]
86		if not self.spam_checker.check_username_for_spam(user)
87		]
	90	non_spammy_users = []
	91	for user in results["results"]:
	92	if not await self.spam_checker.check_username_for_spam(user):
	93	non_spammy_users.append(user)
	94	results["results"] = non_spammy_users
88	95
89	96	return results
90	97
91		def notify_new_event(self):
	98	def notify_new_event(self) -> None:
92	99	"""Called when there may be more deltas to process
93	100	"""
94	101	if not self.update_user_directory:

106	113	self._is_processing = True
107	114	run_as_background_process("user_directory.notify_new_event", process)
108	115
109		async def handle_local_profile_change(self, user_id, profile):
	116	async def handle_local_profile_change(
	117	self, user_id: str, profile: ProfileInfo
	118	) -> None:
110	119	"""Called to update index of our local user profiles when they change
111	120	irrespective of any rooms the user may be in.
112	121	"""
113	122	# FIXME(#3714): We should probably do this in the same worker as all
114	123	# the other changes.
	124
	125	# Support users are for diagnostics and should not appear in the user directory.
115	126	is_support = await self.store.is_support_user(user_id)
116		# Support users are for diagnostics and should not appear in the user directory.
117		if not is_support:
	127	# When change profile information of deactivated user it should not appear in the user directory.
	128	is_deactivated = await self.store.get_user_deactivated_status(user_id)
	129
	130	if not (is_support or is_deactivated):
118	131	await self.store.update_profile_in_user_dir(
119	132	user_id, profile.display_name, profile.avatar_url
120	133	)
121	134
122		async def handle_user_deactivated(self, user_id):
	135	async def handle_user_deactivated(self, user_id: str) -> None:
123	136	"""Called when a user ID is deactivated
124	137	"""
125	138	# FIXME(#3714): We should probably do this in the same worker as all
126	139	# the other changes.
127	140	await self.store.remove_from_user_dir(user_id)
128	141
129		async def _unsafe_process(self):
	142	async def _unsafe_process(self) -> None:
130	143	# If self.pos is None then means we haven't fetched it from DB
131	144	if self.pos is None:
132	145	self.pos = await self.store.get_user_directory_stream_pos()

161	174
162	175	await self.store.update_user_directory_stream_pos(max_pos)
163	176
164		async def _handle_deltas(self, deltas):
	177	async def _handle_deltas(self, deltas: List[Dict[str, Any]]) -> None:
165	178	"""Called with the state deltas to process
166	179	"""
167	180	for delta in deltas:

231	244	logger.debug("Ignoring irrelevant type: %r", typ)
232	245
233	246	async def _handle_room_publicity_change(
234		self, room_id, prev_event_id, event_id, typ
235		):
	247	self,
	248	room_id: str,
	249	prev_event_id: Optional[str],
	250	event_id: Optional[str],
	251	typ: str,
	252	) -> None:
236	253	"""Handle a room having potentially changed from/to world_readable/publicly
237	254	joinable.
238	255
239	256	Args:
240		room_id (str)
241		prev_event_id (str\|None): The previous event before the state change
242		event_id (str\|None): The new event after the state change
243		typ (str): Type of the event
	257	room_id: The ID of the room which changed.
	258	prev_event_id: The previous event before the state change
	259	event_id: The new event after the state change
	260	typ: Type of the event
244	261	"""
245	262	logger.debug("Handling change for %s: %s", typ, room_id)
246	263

249	266	prev_event_id,
250	267	event_id,
251	268	key_name="history_visibility",
252		public_value="world_readable",
	269	public_value=HistoryVisibility.WORLD_READABLE,
253	270	)
254	271	elif typ == EventTypes.JoinRules:
255	272	change = await self._get_key_change(

298	315	for user_id, profile in users_with_profile.items():
299	316	await self._handle_new_user(room_id, user_id, profile)
300	317
301		async def _handle_new_user(self, room_id, user_id, profile):
	318	async def _handle_new_user(
	319	self, room_id: str, user_id: str, profile: ProfileInfo
	320	) -> None:
302	321	"""Called when we might need to add user to directory
303	322
304	323	Args:
305		room_id (str): room_id that user joined or started being public
306		user_id (str)
	324	room_id: The room ID that user joined or started being public
	325	user_id
307	326	"""
308	327	logger.debug("Adding new user to dir, %r", user_id)
309	328

351	370	if to_insert:
352	371	await self.store.add_users_who_share_private_room(room_id, to_insert)
353	372
354		async def _handle_remove_user(self, room_id, user_id):
	373	async def _handle_remove_user(self, room_id: str, user_id: str) -> None:
355	374	"""Called when we might need to remove user from directory
356	375
357	376	Args:
358		room_id (str): room_id that user left or stopped being public that
359		user_id (str)
	377	room_id: The room ID that user left or stopped being public that
	378	user_id
360	379	"""
361	380	logger.debug("Removing user %r", user_id)
362	381

369	388	if len(rooms_user_is_in) == 0:
370	389	await self.store.remove_from_user_dir(user_id)
371	390
372		async def _handle_profile_change(self, user_id, room_id, prev_event_id, event_id):
	391	async def _handle_profile_change(
	392	self,
	393	user_id: str,
	394	room_id: str,
	395	prev_event_id: Optional[str],
	396	event_id: Optional[str],
	397	) -> None:
373	398	"""Check member event changes for any profile changes and update the
374	399	database if there are.
375	400	"""

+51

-28

synapse/http/client.py less more

124	124	return _scheduler
125	125
126	126
127		class IPBlacklistingResolver:
	127	class _IPBlacklistingResolver:
128	128	"""
129	129	A proxy for reactor.nameResolver which only produces non-blacklisted IP
130	130	addresses, preventing DNS rebinding attacks on URL preview.

196	196	)
197	197
198	198	return r
	199
	200
	201	@implementer(IReactorPluggableNameResolver)
	202	class BlacklistingReactorWrapper:
	203	"""
	204	A Reactor wrapper which will prevent DNS resolution to blacklisted IP
	205	addresses, to prevent DNS rebinding.
	206	"""
	207
	208	def __init__(
	209	self,
	210	reactor: IReactorPluggableNameResolver,
	211	ip_whitelist: Optional[IPSet],
	212	ip_blacklist: IPSet,
	213	):
	214	self._reactor = reactor
	215
	216	# We need to use a DNS resolver which filters out blacklisted IP
	217	# addresses, to prevent DNS rebinding.
	218	self._nameResolver = _IPBlacklistingResolver(
	219	self._reactor, ip_whitelist, ip_blacklist
	220	)
	221
	222	def __getattr__(self, attr: str) -> Any:
	223	# Passthrough to the real reactor except for the DNS resolver.
	224	if attr == "nameResolver":
	225	return self._nameResolver
	226	else:
	227	return getattr(self._reactor, attr)
199	228
200	229
201	230	class BlacklistingAgentWrapper(Agent):

291	320	self.user_agent = self.user_agent.encode("ascii")
292	321
293	322	if self._ip_blacklist:
294		real_reactor = hs.get_reactor()
295	323	# If we have an IP blacklist, we need to use a DNS resolver which
296	324	# filters out blacklisted IP addresses, to prevent DNS rebinding.
297		nameResolver = IPBlacklistingResolver(
298		real_reactor, self._ip_whitelist, self._ip_blacklist
	325	self.reactor = BlacklistingReactorWrapper(
	326	hs.get_reactor(), self._ip_whitelist, self._ip_blacklist
299	327	)
300
301		@implementer(IReactorPluggableNameResolver)
302		class Reactor:
303		def __getattr__(_self, attr):
304		if attr == "nameResolver":
305		return nameResolver
306		else:
307		return getattr(real_reactor, attr)
308
309		self.reactor = Reactor()
310	328	else:
311	329	self.reactor = hs.get_reactor()
312	330

322	340
323	341	self.agent = ProxyAgent(
324	342	self.reactor,
	343	hs.get_reactor(),
325	344	connectTimeout=15,
326	345	contextFactory=self.hs.get_http_client_context_factory(),
327	346	pool=pool,

701	720
702	721	try:
703	722	length = await make_deferred_yieldable(
704		readBodyToFile(response, output_stream, max_size)
	723	read_body_with_max_size(response, output_stream, max_size)
705	724	)
706		except SynapseError:
707		# This can happen e.g. because the body is too large.
708		raise
	725	except BodyExceededMaxSize:
	726	SynapseError(
	727	502,
	728	"Requested file is too large > %r bytes" % (max_size,),
	729	Codes.TOO_LARGE,
	730	)
709	731	except Exception as e:
710	732	raise SynapseError(502, ("Failed to download remote body: %s" % e)) from e
711	733

729	751	return f
730	752
731	753
732		class _ReadBodyToFileProtocol(protocol.Protocol):
	754	class BodyExceededMaxSize(Exception):
	755	"""The maximum allowed size of the HTTP body was exceeded."""
	756
	757
	758	class _ReadBodyWithMaxSizeProtocol(protocol.Protocol):
733	759	def __init__(
734	760	self, stream: BinaryIO, deferred: defer.Deferred, max_size: Optional[int]
735	761	):

742	768	self.stream.write(data)
743	769	self.length += len(data)
744	770	if self.max_size is not None and self.length >= self.max_size:
745		self.deferred.errback(
746		SynapseError(
747		502,
748		"Requested file is too large > %r bytes" % (self.max_size,),
749		Codes.TOO_LARGE,
750		)
751		)
	771	self.deferred.errback(BodyExceededMaxSize())
752	772	self.deferred = defer.Deferred()
753	773	self.transport.loseConnection()
754	774

763	783	self.deferred.errback(reason)
764	784
765	785
766		def readBodyToFile(
	786	def read_body_with_max_size(
767	787	response: IResponse, stream: BinaryIO, max_size: Optional[int]
768	788	) -> defer.Deferred:
769	789	"""
770	790	Read a HTTP response body to a file-object. Optionally enforcing a maximum file size.
	791
	792	If the maximum file size is reached, the returned Deferred will resolve to a
	793	Failure with a BodyExceededMaxSize exception.
771	794
772	795	Args:
773	796	response: The HTTP response to read from.

779	802	"""
780	803
781	804	d = defer.Deferred()
782		response.deliverBody(_ReadBodyToFileProtocol(stream, d, max_size))
	805	response.deliverBody(_ReadBodyWithMaxSizeProtocol(stream, d, max_size))
783	806	return d
784	807
785	808

+12

-4

synapse/http/federation/matrix_federation_agent.py less more

15	15	import urllib.parse
16	16	from typing import List, Optional
17	17
18		from netaddr import AddrFormatError, IPAddress
	18	from netaddr import AddrFormatError, IPAddress, IPSet
19	19	from zope.interface import implementer
20	20
21	21	from twisted.internet import defer

30	30	from twisted.web.iweb import IAgent, IAgentEndpointFactory, IBodyProducer
31	31
32	32	from synapse.crypto.context_factory import FederationPolicyForHTTPS
	33	from synapse.http.client import BlacklistingAgentWrapper
33	34	from synapse.http.federation.srv_resolver import Server, SrvResolver
34	35	from synapse.http.federation.well_known_resolver import WellKnownResolver
35	36	from synapse.logging.context import make_deferred_yieldable, run_in_background

69	70	reactor: IReactorCore,
70	71	tls_client_options_factory: Optional[FederationPolicyForHTTPS],
71	72	user_agent: bytes,
	73	ip_blacklist: IPSet,
72	74	_srv_resolver: Optional[SrvResolver] = None,
73	75	_well_known_resolver: Optional[WellKnownResolver] = None,
74	76	):

89	91	self.user_agent = user_agent
90	92
91	93	if _well_known_resolver is None:
	94	# Note that the name resolver has already been wrapped in a
	95	# IPBlacklistingResolver by MatrixFederationHttpClient.
92	96	_well_known_resolver = WellKnownResolver(
93	97	self._reactor,
94		agent=Agent(
	98	agent=BlacklistingAgentWrapper(
	99	Agent(
	100	self._reactor,
	101	pool=self._pool,
	102	contextFactory=tls_client_options_factory,
	103	),
95	104	self._reactor,
96		pool=self._pool,
97		contextFactory=tls_client_options_factory,
	105	ip_blacklist=ip_blacklist,
98	106	),
99	107	user_agent=self.user_agent,
100	108	)

+23

-2

synapse/http/federation/well_known_resolver.py less more

14	14	import logging
15	15	import random
16	16	import time
	17	from io import BytesIO
17	18	from typing import Callable, Dict, Optional, Tuple
18	19
19	20	import attr
20	21
21	22	from twisted.internet import defer
22	23	from twisted.internet.interfaces import IReactorTime
23		from twisted.web.client import RedirectAgent, readBody
	24	from twisted.web.client import RedirectAgent
24	25	from twisted.web.http import stringToDatetime
25	26	from twisted.web.http_headers import Headers
26	27	from twisted.web.iweb import IAgent, IResponse
27	28
	29	from synapse.http.client import BodyExceededMaxSize, read_body_with_max_size
28	30	from synapse.logging.context import make_deferred_yieldable
29	31	from synapse.util import Clock, json_decoder
30	32	from synapse.util.caches.ttlcache import TTLCache

51	53
52	54	# lower bound for .well-known cache period
53	55	WELL_KNOWN_MIN_CACHE_PERIOD = 5 * 60
	56
	57	# The maximum size (in bytes) to allow a well-known file to be.
	58	WELL_KNOWN_MAX_SIZE = 50 * 1024 # 50 KiB
54	59
55	60	# Attempt to refetch a cached well-known N% of the TTL before it expires.
56	61	# e.g. if set to 0.2 and we have a cached entry with a TTL of 5mins, then

228	233	server_name: name of the server, from the requested url
229	234	retry: Whether to retry the request if it fails.
230	235
	236	Raises:
	237	_FetchWellKnownFailure if we fail to lookup a result
	238
231	239	Returns:
232	240	Returns the response object and body. Response may be a non-200 response.
233	241	"""

249	257	b"GET", uri, headers=Headers(headers)
250	258	)
251	259	)
252		body = await make_deferred_yieldable(readBody(response))
	260	body_stream = BytesIO()
	261	await make_deferred_yieldable(
	262	read_body_with_max_size(response, body_stream, WELL_KNOWN_MAX_SIZE)
	263	)
	264	body = body_stream.getvalue()
253	265
254	266	if 500 <= response.code < 600:
255	267	raise Exception("Non-200 response %s" % (response.code,))

258	270	except defer.CancelledError:
259	271	# Bail if we've been cancelled
260	272	raise
	273	except BodyExceededMaxSize:
	274	# If the well-known file was too large, do not keep attempting
	275	# to download it, but consider it a temporary error.
	276	logger.warning(
	277	"Requested .well-known file for %s is too large > %r bytes",
	278	server_name.decode("ascii"),
	279	WELL_KNOWN_MAX_SIZE,
	280	)
	281	raise _FetchWellKnownFailure(temporary=True)
261	282	except Exception as e:
262	283	if not retry or i >= WELL_KNOWN_RETRY_ATTEMPTS:
263	284	logger.info("Error fetching %s: %s", uri_str, e)

+20

-21

synapse/http/matrixfederationclient.py less more

25	25	from canonicaljson import encode_canonical_json
26	26	from prometheus_client import Counter
27	27	from signedjson.sign import sign_json
28		from zope.interface import implementer
29	28
30	29	from twisted.internet import defer
31	30	from twisted.internet.error import DNSLookupError
32		from twisted.internet.interfaces import IReactorPluggableNameResolver, IReactorTime
	31	from twisted.internet.interfaces import IReactorTime
33	32	from twisted.internet.task import _EPSILON, Cooperator
34	33	from twisted.web.http_headers import Headers
35	34	from twisted.web.iweb import IBodyProducer, IResponse

37	36	import synapse.metrics
38	37	import synapse.util.retryutils
39	38	from synapse.api.errors import (
	39	Codes,
40	40	FederationDeniedError,
41	41	HttpResponseException,
42	42	RequestSendFailed,
	43	SynapseError,
43	44	)
44	45	from synapse.http import QuieterFileBodyProducer
45	46	from synapse.http.client import (
46	47	BlacklistingAgentWrapper,
47		IPBlacklistingResolver,
	48	BlacklistingReactorWrapper,
	49	BodyExceededMaxSize,
48	50	encode_query_args,
49		readBodyToFile,
	51	read_body_with_max_size,
50	52	)
51	53	from synapse.http.federation.matrix_federation_agent import MatrixFederationAgent
52	54	from synapse.logging.context import make_deferred_yieldable

220	222	self.signing_key = hs.signing_key
221	223	self.server_name = hs.hostname
222	224
223		real_reactor = hs.get_reactor()
224
225	225	# We need to use a DNS resolver which filters out blacklisted IP
226	226	# addresses, to prevent DNS rebinding.
227		nameResolver = IPBlacklistingResolver(
228		real_reactor, None, hs.config.federation_ip_range_blacklist
229		)
230
231		@implementer(IReactorPluggableNameResolver)
232		class Reactor:
233		def __getattr__(_self, attr):
234		if attr == "nameResolver":
235		return nameResolver
236		else:
237		return getattr(real_reactor, attr)
238
239		self.reactor = Reactor()
	227	self.reactor = BlacklistingReactorWrapper(
	228	hs.get_reactor(), None, hs.config.federation_ip_range_blacklist
	229	)
240	230
241	231	user_agent = hs.version_string
242	232	if hs.config.user_agent_suffix:

244	234	user_agent = user_agent.encode("ascii")
245	235
246	236	self.agent = MatrixFederationAgent(
247		self.reactor, tls_client_options_factory, user_agent
	237	self.reactor,
	238	tls_client_options_factory,
	239	user_agent,
	240	hs.config.federation_ip_range_blacklist,
248	241	)
249	242
250	243	# Use a BlacklistingAgentWrapper to prevent circumventing the IP

984	977	headers = dict(response.headers.getAllRawHeaders())
985	978
986	979	try:
987		d = readBodyToFile(response, output_stream, max_size)
	980	d = read_body_with_max_size(response, output_stream, max_size)
988	981	d.addTimeout(self.default_timeout, self.reactor)
989	982	length = await make_deferred_yieldable(d)
	983	except BodyExceededMaxSize:
	984	msg = "Requested file is too large > %r bytes" % (max_size,)
	985	logger.warning(
	986	"{%s} [%s] %s", request.txn_id, request.destination, msg,
	987	)
	988	SynapseError(502, msg, Codes.TOO_LARGE)
990	989	except Exception as e:
991	990	logger.warning(
992	991	"{%s} [%s] Error reading response: %s",

+13

-3

synapse/http/proxyagent.py less more

38	38	reactor: twisted reactor to place outgoing
39	39	connections.
40	40
	41	proxy_reactor: twisted reactor to use for connections to the proxy server
	42	reactor might have some blacklisting applied (i.e. for DNS queries),
	43	but we need unblocked access to the proxy.
	44
41	45	contextFactory (IPolicyForHTTPS): A factory for TLS contexts, to control the
42	46	verification parameters of OpenSSL. The default is to use a
43	47	`BrowserLikePolicyForHTTPS`, so unless you have special

58	62	def __init__(
59	63	self,
60	64	reactor,
	65	proxy_reactor=None,
61	66	contextFactory=BrowserLikePolicyForHTTPS(),
62	67	connectTimeout=None,
63	68	bindAddress=None,

67	72	):
68	73	_AgentBase.__init__(self, reactor, pool)
69	74
	75	if proxy_reactor is None:
	76	self.proxy_reactor = reactor
	77	else:
	78	self.proxy_reactor = proxy_reactor
	79
70	80	self._endpoint_kwargs = {}
71	81	if connectTimeout is not None:
72	82	self._endpoint_kwargs["timeout"] = connectTimeout

74	84	self._endpoint_kwargs["bindAddress"] = bindAddress
75	85
76	86	self.http_proxy_endpoint = _http_proxy_endpoint(
77		http_proxy, reactor, **self._endpoint_kwargs
	87	http_proxy, self.proxy_reactor, **self._endpoint_kwargs
78	88	)
79	89
80	90	self.https_proxy_endpoint = _http_proxy_endpoint(
81		https_proxy, reactor, **self._endpoint_kwargs
	91	https_proxy, self.proxy_reactor, **self._endpoint_kwargs
82	92	)
83	93
84	94	self._policy_for_https = contextFactory

136	146	request_path = uri
137	147	elif parsed_uri.scheme == b"https" and self.https_proxy_endpoint:
138	148	endpoint = HTTPConnectProxyEndpoint(
139		self._reactor,
	149	self.proxy_reactor,
140	150	self.https_proxy_endpoint,
141	151	parsed_uri.host,
142	152	parsed_uri.port,

+5

-3

synapse/http/server.py less more

274	274	formatting responses and errors as JSON.
275	275	"""
276	276
	277	def __init__(self, canonical_json=False, extract_context=False):
	278	super().__init__(extract_context)
	279	self.canonical_json = canonical_json
	280
277	281	def _send_response(
278	282	self, request: Request, code: int, response_object: Any,
279	283	):

317	321	)
318	322
319	323	def __init__(self, hs, canonical_json=True, extract_context=False):
320		super().__init__(extract_context)
321
322		self.canonical_json = canonical_json
	324	super().__init__(canonical_json, extract_context)
323	325	self.clock = hs.get_clock()
324	326	self.path_regexs = {}
325	327	self.hs = hs

+1

-2

synapse/http/site.py less more

127	127
128	128	# create a LogContext for this request
129	129	request_id = self.get_request_id()
130		logcontext = self.logcontext = LoggingContext(request_id)
131		logcontext.request = request_id
	130	self.logcontext = LoggingContext(request_id, request=request_id)
132	131
133	132	# override the Server header which is set by twisted
134	133	self.setHeader("Server", self.site.server_version_string)

+7

-21

synapse/logging/context.py less more

202	202	def copy_to(self, record):
203	203	pass
204	204
205		def copy_to_twisted_log_entry(self, record):
206		record["request"] = None
207		record["scope"] = None
208
209	205	def start(self, rusage: "Optional[resource._RUsage]"):
210	206	pass
211	207

371	367	# we also track the current scope:
372	368	record.scope = self.scope
373	369
374		def copy_to_twisted_log_entry(self, record) -> None:
375		"""
376		Copy logging fields from this context to a Twisted log record.
377		"""
378		record["request"] = self.request
379		record["scope"] = self.scope
380
381	370	def start(self, rusage: "Optional[resource._RUsage]") -> None:
382	371	"""
383	372	Record that this logcontext is currently running.

541	530	class LoggingContextFilter(logging.Filter):
542	531	"""Logging filter that adds values from the current logging context to each
543	532	record.
544		Args:
545		**defaults: Default values to avoid formatters complaining about
546		missing fields
547		"""
548
549		def __init__(self, **defaults) -> None:
550		self.defaults = defaults
	533	"""
	534
	535	def __init__(self, request: str = ""):
	536	self._default_request = request
551	537
552	538	def filter(self, record) -> Literal[True]:
553	539	"""Add each fields from the logging contexts to the record.

555	541	True to include the record in the log output.
556	542	"""
557	543	context = current_context()
558		for key, value in self.defaults.items():
559		setattr(record, key, value)
	544	record.request = self._default_request
560	545
561	546	# context should never be None, but if it somehow ends up being, then
562	547	# we end up in a death spiral of infinite loops, so let's check, for
563	548	# robustness' sake.
564	549	if context is not None:
565		context.copy_to(record)
	550	# Logging is interested in the request.
	551	record.request = context.request
566	552
567	553	return True
568	554

+5

-11

synapse/metrics/background_process_metrics.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import inspect
16	15	import logging
17	16	import threading
18	17	from functools import wraps

24	23
25	24	from synapse.logging.context import LoggingContext, PreserveLoggingContext
26	25	from synapse.logging.opentracing import noop_context_manager, start_active_span
	26	from synapse.util.async_helpers import maybe_awaitable
27	27
28	28	if TYPE_CHECKING:
29	29	import resource

198	198	_background_process_start_count.labels(desc).inc()
199	199	_background_process_in_flight_count.labels(desc).inc()
200	200
201		with BackgroundProcessLoggingContext(desc) as context:
202		context.request = "%s-%i" % (desc, count)
	201	with BackgroundProcessLoggingContext(desc, "%s-%i" % (desc, count)) as context:
203	202	try:
204	203	ctx = noop_context_manager()
205	204	if bg_start_span:
206	205	ctx = start_active_span(desc, tags={"request_id": context.request})
207	206	with ctx:
208		result = func(args, *kwargs)
209
210		if inspect.isawaitable(result):
211		result = await result
212
213		return result
	207	return await maybe_awaitable(func(args, *kwargs))
214	208	except Exception:
215	209	logger.exception(
216	210	"Background process '%s' threw an exception", desc,

248	242
249	243	__slots__ = ["_proc"]
250	244
251		def __init__(self, name: str):
252		super().__init__(name)
	245	def __init__(self, name: str, request: Optional[str] = None):
	246	super().__init__(name, request=request)
253	247
254	248	self._proc = _BackgroundProcess(name, self)
255	249

+4

-2

synapse/notifier.py less more

33	33	from twisted.internet import defer
34	34
35	35	import synapse.server
36		from synapse.api.constants import EventTypes, Membership
	36	from synapse.api.constants import EventTypes, HistoryVisibility, Membership
37	37	from synapse.api.errors import AuthError
38	38	from synapse.events import EventBase
39	39	from synapse.handlers.presence import format_user_presence_state

610	610	room_id, EventTypes.RoomHistoryVisibility, ""
611	611	)
612	612	if state and "history_visibility" in state.content:
613		return state.content["history_visibility"] == "world_readable"
	613	return (
	614	state.content["history_visibility"] == HistoryVisibility.WORLD_READABLE
	615	)
614	616	else:
615	617	return False
616	618

+106

-2

synapse/push/__init__.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
	15	import abc
	16	from typing import TYPE_CHECKING, Any, Dict, Optional
	17
	18	import attr
	19
	20	from synapse.types import JsonDict, RoomStreamToken
	21
	22	if TYPE_CHECKING:
	23	from synapse.app.homeserver import HomeServer
	24
	25
	26	@attr.s(slots=True)
	27	class PusherConfig:
	28	"""Parameters necessary to configure a pusher."""
	29
	30	id = attr.ib(type=Optional[str])
	31	user_name = attr.ib(type=str)
	32	access_token = attr.ib(type=Optional[int])
	33	profile_tag = attr.ib(type=str)
	34	kind = attr.ib(type=str)
	35	app_id = attr.ib(type=str)
	36	app_display_name = attr.ib(type=str)
	37	device_display_name = attr.ib(type=str)
	38	pushkey = attr.ib(type=str)
	39	ts = attr.ib(type=int)
	40	lang = attr.ib(type=Optional[str])
	41	data = attr.ib(type=Optional[JsonDict])
	42	last_stream_ordering = attr.ib(type=int)
	43	last_success = attr.ib(type=Optional[int])
	44	failing_since = attr.ib(type=Optional[int])
	45
	46	def as_dict(self) -> Dict[str, Any]:
	47	"""Information that can be retrieved about a pusher after creation."""
	48	return {
	49	"app_display_name": self.app_display_name,
	50	"app_id": self.app_id,
	51	"data": self.data,
	52	"device_display_name": self.device_display_name,
	53	"kind": self.kind,
	54	"lang": self.lang,
	55	"profile_tag": self.profile_tag,
	56	"pushkey": self.pushkey,
	57	}
	58
	59
	60	@attr.s(slots=True)
	61	class ThrottleParams:
	62	"""Parameters for controlling the rate of sending pushes via email."""
	63
	64	last_sent_ts = attr.ib(type=int)
	65	throttle_ms = attr.ib(type=int)
	66
	67
	68	class Pusher(metaclass=abc.ABCMeta):
	69	def __init__(self, hs: "HomeServer", pusher_config: PusherConfig):
	70	self.hs = hs
	71	self.store = self.hs.get_datastore()
	72	self.clock = self.hs.get_clock()
	73
	74	self.pusher_id = pusher_config.id
	75	self.user_id = pusher_config.user_name
	76	self.app_id = pusher_config.app_id
	77	self.pushkey = pusher_config.pushkey
	78
	79	self.last_stream_ordering = pusher_config.last_stream_ordering
	80
	81	# This is the highest stream ordering we know it's safe to process.
	82	# When new events arrive, we'll be given a window of new events: we
	83	# should honour this rather than just looking for anything higher
	84	# because of potential out-of-order event serialisation.
	85	self.max_stream_ordering = self.store.get_room_max_stream_ordering()
	86
	87	def on_new_notifications(self, max_token: RoomStreamToken) -> None:
	88	# We just use the minimum stream ordering and ignore the vector clock
	89	# component. This is safe to do as long as we always ignore the vector
	90	# clock components.
	91	max_stream_ordering = max_token.stream
	92
	93	self.max_stream_ordering = max(max_stream_ordering, self.max_stream_ordering)
	94	self._start_processing()
	95
	96	@abc.abstractmethod
	97	def _start_processing(self):
	98	"""Start processing push notifications."""
	99	raise NotImplementedError()
	100
	101	@abc.abstractmethod
	102	def on_new_receipts(self, min_stream_id: int, max_stream_id: int) -> None:
	103	raise NotImplementedError()
	104
	105	@abc.abstractmethod
	106	def on_started(self, have_notifs: bool) -> None:
	107	"""Called when this pusher has been started.
	108
	109	Args:
	110	should_check_for_notifs: Whether we should immediately
	111	check for push to send. Set to False only if it's known there
	112	is nothing to send
	113	"""
	114	raise NotImplementedError()
	115
	116	@abc.abstractmethod
	117	def on_stop(self) -> None:
	118	raise NotImplementedError()
	119
15	120
16	121	class PusherConfigException(Exception):
17		def __init__(self, msg):
18		super().__init__(msg)
	122	"""An error occurred when creating a pusher."""

+10

-5

synapse/push/action_generator.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import TYPE_CHECKING
16	17
	18	from synapse.events import EventBase
	19	from synapse.events.snapshot import EventContext
	20	from synapse.push.bulk_push_rule_evaluator import BulkPushRuleEvaluator
17	21	from synapse.util.metrics import Measure
18	22
19		from .bulk_push_rule_evaluator import BulkPushRuleEvaluator
	23	if TYPE_CHECKING:
	24	from synapse.app.homeserver import HomeServer
20	25
21	26	logger = logging.getLogger(__name__)
22	27
23	28
24	29	class ActionGenerator:
25		def __init__(self, hs):
26		self.hs = hs
	30	def __init__(self, hs: "HomeServer"):
27	31	self.clock = hs.get_clock()
28		self.store = hs.get_datastore()
29	32	self.bulk_evaluator = BulkPushRuleEvaluator(hs)
30	33	# really we want to get all user ids and all profile tags too,
31	34	# since we want the actions for each profile tag for every user and

34	37	# event stream, so we just run the rules for a client with no profile
35	38	# tag (ie. we just need all the users).
36	39
37		async def handle_push_actions_for_event(self, event, context):
	40	async def handle_push_actions_for_event(
	41	self, event: EventBase, context: EventContext
	42	) -> None:
38	43	with Measure(self.clock, "action_for_event_by_user"):
39	44	await self.bulk_evaluator.action_for_event_by_user(event, context)

+18

-5

synapse/push/baserules.py less more

14	14	# limitations under the License.
15	15
16	16	import copy
	17	from typing import Any, Dict, List
17	18
18	19	from synapse.push.rulekinds import PRIORITY_CLASS_INVERSE_MAP, PRIORITY_CLASS_MAP
19	20
20	21
21		def list_with_base_rules(rawrules, use_new_defaults=False):
	22	def list_with_base_rules(
	23	rawrules: List[Dict[str, Any]], use_new_defaults: bool = False
	24	) -> List[Dict[str, Any]]:
22	25	"""Combine the list of rules set by the user with the default push rules
23	26
24	27	Args:
25		rawrules(list): The rules the user has modified or set.
26		use_new_defaults(bool): Whether to use the new experimental default rules when
	28	rawrules: The rules the user has modified or set.
	29	use_new_defaults: Whether to use the new experimental default rules when
27	30	appending or prepending default rules.
28	31
29	32	Returns:

93	96	return ruleslist
94	97
95	98
96		def make_base_append_rules(kind, modified_base_rules, use_new_defaults=False):
	99	def make_base_append_rules(
	100	kind: str,
	101	modified_base_rules: Dict[str, Dict[str, Any]],
	102	use_new_defaults: bool = False,
	103	) -> List[Dict[str, Any]]:
97	104	rules = []
98	105
99	106	if kind == "override":

115	122	rules = copy.deepcopy(rules)
116	123	for r in rules:
117	124	# Only modify the actions, keep the conditions the same.
	125	assert isinstance(r["rule_id"], str)
118	126	modified = modified_base_rules.get(r["rule_id"])
119	127	if modified:
120	128	r["actions"] = modified["actions"]

122	130	return rules
123	131
124	132
125		def make_base_prepend_rules(kind, modified_base_rules, use_new_defaults=False):
	133	def make_base_prepend_rules(
	134	kind: str,
	135	modified_base_rules: Dict[str, Dict[str, Any]],
	136	use_new_defaults: bool = False,
	137	) -> List[Dict[str, Any]]:
126	138	rules = []
127	139
128	140	if kind == "override":

132	144	rules = copy.deepcopy(rules)
133	145	for r in rules:
134	146	# Only modify the actions, keep the conditions the same.
	147	assert isinstance(r["rule_id"], str)
135	148	modified = modified_base_rules.get(r["rule_id"])
136	149	if modified:
137	150	r["actions"] = modified["actions"]

+61

-37

synapse/push/bulk_push_rule_evaluator.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
	17	from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Union
17	18
18	19	import attr
19	20	from prometheus_client import Counter

24	25	from synapse.events.snapshot import EventContext
25	26	from synapse.state import POWER_KEY
26	27	from synapse.util.async_helpers import Linearizer
27		from synapse.util.caches import register_cache
	28	from synapse.util.caches import CacheMetric, register_cache
28	29	from synapse.util.caches.descriptors import lru_cache
29	30	from synapse.util.caches.lrucache import LruCache
30	31
31	32	from .push_rule_evaluator import PushRuleEvaluatorForEvent
32	33
	34	if TYPE_CHECKING:
	35	from synapse.app.homeserver import HomeServer
	36
33	37	logger = logging.getLogger(__name__)
34
35
36		rules_by_room = {}
37	38
38	39
39	40	push_rules_invalidation_counter = Counter(

100	101	room at once.
101	102	"""
102	103
103		def __init__(self, hs):
	104	def __init__(self, hs: "HomeServer"):
104	105	self.hs = hs
105	106	self.store = hs.get_datastore()
106	107	self.auth = hs.get_auth()

112	113	resizable=False,
113	114	)
114	115
115		async def _get_rules_for_event(self, event, context):
	116	async def _get_rules_for_event(
	117	self, event: EventBase, context: EventContext
	118	) -> Dict[str, List[Dict[str, Any]]]:
116	119	"""This gets the rules for all users in the room at the time of the event,
117	120	as well as the push rules for the invitee if the event is an invite.
118	121

139	142	return rules_by_user
140	143
141	144	@lru_cache()
142		def _get_rules_for_room(self, room_id):
	145	def _get_rules_for_room(self, room_id: str) -> "RulesForRoom":
143	146	"""Get the current RulesForRoom object for the given room id
144
145		Returns:
146		RulesForRoom
147	147	"""
148	148	# It's important that RulesForRoom gets added to self._get_rules_for_room.cache
149	149	# before any lookup methods get called on it as otherwise there may be

155	155	self.room_push_rule_cache_metrics,
156	156	)
157	157
158		async def _get_power_levels_and_sender_level(self, event, context):
	158	async def _get_power_levels_and_sender_level(
	159	self, event: EventBase, context: EventContext
	160	) -> Tuple[dict, int]:
159	161	prev_state_ids = await context.get_prev_state_ids()
160	162	pl_event_id = prev_state_ids.get(POWER_KEY)
161	163	if pl_event_id:
162	164	# fastpath: if there's a power level event, that's all we need, and
163	165	# not having a power level event is an extreme edge case
164		pl_event = await self.store.get_event(pl_event_id)
165		auth_events = {POWER_KEY: pl_event}
	166	auth_events = {POWER_KEY: await self.store.get_event(pl_event_id)}
166	167	else:
167	168	auth_events_ids = self.auth.compute_auth_events(
168	169	event, prev_state_ids, for_verification=False
169	170	)
170		auth_events = await self.store.get_events(auth_events_ids)
171		auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
	171	auth_events_dict = await self.store.get_events(auth_events_ids)
	172	auth_events = {(e.type, e.state_key): e for e in auth_events_dict.values()}
172	173
173	174	sender_level = get_user_power_level(event.sender, auth_events)
174	175

176	177
177	178	return pl_event.content if pl_event else {}, sender_level
178	179
179		async def action_for_event_by_user(self, event, context) -> None:
	180	async def action_for_event_by_user(
	181	self, event: EventBase, context: EventContext
	182	) -> None:
180	183	"""Given an event and context, evaluate the push rules, check if the message
181	184	should increment the unread count, and insert the results into the
182	185	event_push_actions_staging table.

184	187	count_as_unread = _should_count_as_unread(event, context)
185	188
186	189	rules_by_user = await self._get_rules_for_event(event, context)
187		actions_by_user = {}
	190	actions_by_user = {} # type: Dict[str, List[Union[dict, str]]]
188	191
189	192	room_members = await self.store.get_joined_users_from_context(event, context)
190	193

197	200	event, len(room_members), sender_power_level, power_levels
198	201	)
199	202
200		condition_cache = {}
	203	condition_cache = {} # type: Dict[str, bool]
201	204
202	205	for uid, rules in rules_by_user.items():
203	206	if event.sender == uid:

248	251	)
249	252
250	253
251		def _condition_checker(evaluator, conditions, uid, display_name, cache):
	254	def _condition_checker(
	255	evaluator: PushRuleEvaluatorForEvent,
	256	conditions: List[dict],
	257	uid: str,
	258	display_name: str,
	259	cache: Dict[str, bool],
	260	) -> bool:
252	261	for cond in conditions:
253	262	_id = cond.get("_id", None)
254	263	if _id:

276	285	"""
277	286
278	287	def __init__(
279		self, hs, room_id, rules_for_room_cache: LruCache, room_push_rule_cache_metrics
	288	self,
	289	hs: "HomeServer",
	290	room_id: str,
	291	rules_for_room_cache: LruCache,
	292	room_push_rule_cache_metrics: CacheMetric,
280	293	):
281	294	"""
282	295	Args:
283		hs (HomeServer)
284		room_id (str)
	296	hs: The HomeServer object.
	297	room_id: The room ID.
285	298	rules_for_room_cache: The cache object that caches these
286	299	RoomsForUser objects.
287		room_push_rule_cache_metrics (CacheMetric)
	300	room_push_rule_cache_metrics: The metrics object
288	301	"""
289	302	self.room_id = room_id
290	303	self.is_mine_id = hs.is_mine_id

293	306
294	307	self.linearizer = Linearizer(name="rules_for_room")
295	308
296		self.member_map = {} # event_id -> (user_id, state)
297		self.rules_by_user = {} # user_id -> rules
	309	# event_id -> (user_id, state)
	310	self.member_map = {} # type: Dict[str, Tuple[str, str]]
	311	# user_id -> rules
	312	self.rules_by_user = {} # type: Dict[str, List[Dict[str, dict]]]
298	313
299	314	# The last state group we updated the caches for. If the state_group of
300	315	# a new event comes along, we know that we can just return the cached

314	329	# calculate push for)
315	330	# These never need to be invalidated as we will never set up push for
316	331	# them.
317		self.uninteresting_user_set = set()
	332	self.uninteresting_user_set = set() # type: Set[str]
318	333
319	334	# We need to be clever on the invalidating caches callbacks, as
320	335	# otherwise the invalidation callback holds a reference to the object,

324	339	# to self around in the callback.
325	340	self.invalidate_all_cb = _Invalidation(rules_for_room_cache, room_id)
326	341
327		async def get_rules(self, event, context):
	342	async def get_rules(
	343	self, event: EventBase, context: EventContext
	344	) -> Dict[str, List[Dict[str, dict]]]:
328	345	"""Given an event context return the rules for all users who are
329	346	currently in the room.
330	347	"""

355	372	else:
356	373	current_state_ids = await context.get_current_state_ids()
357	374	push_rules_delta_state_cache_metric.inc_misses()
	375	# Ensure the state IDs exist.
	376	assert current_state_ids is not None
358	377
359	378	push_rules_state_size_counter.inc(len(current_state_ids))
360	379

419	438	return ret_rules_by_user
420	439
421	440	async def _update_rules_with_member_event_ids(
422		self, ret_rules_by_user, member_event_ids, state_group, event
423		):
	441	self,
	442	ret_rules_by_user: Dict[str, list],
	443	member_event_ids: Dict[str, str],
	444	state_group: Optional[int],
	445	event: EventBase,
	446	) -> None:
424	447	"""Update the partially filled rules_by_user dict by fetching rules for
425	448	any newly joined users in the `member_event_ids` list.
426	449
427	450	Args:
428		ret_rules_by_user (dict): Partiallly filled dict of push rules. Gets
	451	ret_rules_by_user: Partially filled dict of push rules. Gets
429	452	updated with any new rules.
430		member_event_ids (dict): Dict of user id to event id for membership events
	453	member_event_ids: Dict of user id to event id for membership events
431	454	that have happened since the last time we filled rules_by_user
432	455	state_group: The state group we are currently computing push rules
433	456	for. Used when updating the cache.
	457	event: The event we are currently computing push rules for.
434	458	"""
435	459	sequence = self.sequence
436	460

448	472	if logger.isEnabledFor(logging.DEBUG):
449	473	logger.debug("Found members %r: %r", self.room_id, members.values())
450	474
451		user_ids = {
	475	joined_user_ids = {
452	476	user_id
453	477	for user_id, membership in members.values()
454	478	if membership == Membership.JOIN
455	479	}
456	480
457		logger.debug("Joined: %r", user_ids)
	481	logger.debug("Joined: %r", joined_user_ids)
458	482
459	483	# Previously we only considered users with pushers or read receipts in that
460	484	# room. We can't do this anymore because we use push actions to calculate unread
461	485	# counts, which don't rely on the user having pushers or sent a read receipt into
462	486	# the room. Therefore we just need to filter for local users here.
463		user_ids = list(filter(self.is_mine_id, user_ids))
	487	user_ids = list(filter(self.is_mine_id, joined_user_ids))
464	488
465	489	rules_by_user = await self.store.bulk_get_push_rules(
466	490	user_ids, on_invalidate=self.invalidate_all_cb

472	496
473	497	self.update_cache(sequence, members, ret_rules_by_user, state_group)
474	498
475		def invalidate_all(self):
	499	def invalidate_all(self) -> None:
476	500	# Note: Don't hand this function directly to an invalidation callback
477	501	# as it keeps a reference to self and will stop this instance from being
478	502	# GC'd if it gets dropped from the rules_to_user cache. Instead use

484	508	self.rules_by_user = {}
485	509	push_rules_invalidation_counter.inc()
486	510
487		def update_cache(self, sequence, members, rules_by_user, state_group):
	511	def update_cache(self, sequence, members, rules_by_user, state_group) -> None:
488	512	if sequence == self.sequence:
489	513	self.member_map.update(members)
490	514	self.rules_by_user = rules_by_user

505	529	cache = attr.ib(type=LruCache)
506	530	room_id = attr.ib(type=str)
507	531
508		def __call__(self):
	532	def __call__(self) -> None:
509	533	rules = self.cache.get(self.room_id, None, update_metrics=False)
510	534	if rules:
511	535	rules.invalidate_all()

+15

-8

synapse/push/clientformat.py less more

13	13	# limitations under the License.
14	14
15	15	import copy
	16	from typing import Any, Dict, List, Optional
16	17
17	18	from synapse.push.rulekinds import PRIORITY_CLASS_INVERSE_MAP, PRIORITY_CLASS_MAP
	19	from synapse.types import UserID
18	20
19	21
20		def format_push_rules_for_user(user, ruleslist):
	22	def format_push_rules_for_user(user: UserID, ruleslist) -> Dict[str, Dict[str, list]]:
21	23	"""Converts a list of rawrules and a enabled map into nested dictionaries
22	24	to match the Matrix client-server format for push rules"""
23	25
24	26	# We're going to be mutating this a lot, so do a deep copy
25	27	ruleslist = copy.deepcopy(ruleslist)
26	28
27		rules = {"global": {}, "device": {}}
	29	rules = {
	30	"global": {},
	31	"device": {},
	32	} # type: Dict[str, Dict[str, List[Dict[str, Any]]]]
28	33
29	34	rules["global"] = _add_empty_priority_class_arrays(rules["global"])
30	35
31	36	for r in ruleslist:
32		rulearray = None
33
34	37	template_name = _priority_class_to_template_name(r["priority_class"])
35	38
36	39	# Remove internal stuff.

56	59	return rules
57	60
58	61
59		def _add_empty_priority_class_arrays(d):
	62	def _add_empty_priority_class_arrays(d: Dict[str, list]) -> Dict[str, list]:
60	63	for pc in PRIORITY_CLASS_MAP.keys():
61	64	d[pc] = []
62	65	return d
63	66
64	67
65		def _rule_to_template(rule):
	68	def _rule_to_template(rule: Dict[str, Any]) -> Optional[Dict[str, Any]]:
66	69	unscoped_rule_id = None
67	70	if "rule_id" in rule:
68	71	unscoped_rule_id = _rule_id_from_namespaced(rule["rule_id"])

81	84	return None
82	85	templaterule = {"actions": rule["actions"]}
83	86	templaterule["pattern"] = thecond["pattern"]
	87	else:
	88	# This should not be reached unless this function is not kept in sync
	89	# with PRIORITY_CLASS_INVERSE_MAP.
	90	raise ValueError("Unexpected template_name: %s" % (template_name,))
84	91
85	92	if unscoped_rule_id:
86	93	templaterule["rule_id"] = unscoped_rule_id

89	96	return templaterule
90	97
91	98
92		def _rule_id_from_namespaced(in_rule_id):
	99	def _rule_id_from_namespaced(in_rule_id: str) -> str:
93	100	return in_rule_id.split("/")[-1]
94	101
95	102
96		def _priority_class_to_template_name(pc):
	103	def _priority_class_to_template_name(pc: int) -> str:
97	104	return PRIORITY_CLASS_INVERSE_MAP[pc]

+54

-64

synapse/push/emailpusher.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16
	16	from typing import TYPE_CHECKING, Dict, List, Optional
	17
	18	from twisted.internet.base import DelayedCall
17	19	from twisted.internet.error import AlreadyCalled, AlreadyCancelled
18	20
19	21	from synapse.metrics.background_process_metrics import run_as_background_process
20		from synapse.types import RoomStreamToken
	22	from synapse.push import Pusher, PusherConfig, ThrottleParams
	23	from synapse.push.mailer import Mailer
	24
	25	if TYPE_CHECKING:
	26	from synapse.app.homeserver import HomeServer
21	27
22	28	logger = logging.getLogger(__name__)
23	29

45	51	INCLUDE_ALL_UNREAD_NOTIFS = False
46	52
47	53
48		class EmailPusher:
	54	class EmailPusher(Pusher):
49	55	"""
50	56	A pusher that sends email notifications about events (approximately)
51	57	when they happen.

53	59	factor out the common parts
54	60	"""
55	61
56		def __init__(self, hs, pusherdict, mailer):
57		self.hs = hs
	62	def __init__(self, hs: "HomeServer", pusher_config: PusherConfig, mailer: Mailer):
	63	super().__init__(hs, pusher_config)
58	64	self.mailer = mailer
59	65
60	66	self.store = self.hs.get_datastore()
61		self.clock = self.hs.get_clock()
62		self.pusher_id = pusherdict["id"]
63		self.user_id = pusherdict["user_name"]
64		self.app_id = pusherdict["app_id"]
65		self.email = pusherdict["pushkey"]
66		self.last_stream_ordering = pusherdict["last_stream_ordering"]
67		self.timed_call = None
68		self.throttle_params = None
69
70		# See httppusher
71		self.max_stream_ordering = None
	67	self.email = pusher_config.pushkey
	68	self.timed_call = None # type: Optional[DelayedCall]
	69	self.throttle_params = {} # type: Dict[str, ThrottleParams]
	70	self._inited = False
72	71
73	72	self._is_processing = False
74	73
75		def on_started(self, should_check_for_notifs):
	74	def on_started(self, should_check_for_notifs: bool) -> None:
76	75	"""Called when this pusher has been started.
77	76
78	77	Args:
79		should_check_for_notifs (bool): Whether we should immediately
	78	should_check_for_notifs: Whether we should immediately
80	79	check for push to send. Set to False only if it's known there
81	80	is nothing to send
82	81	"""
83	82	if should_check_for_notifs and self.mailer is not None:
84	83	self._start_processing()
85	84
86		def on_stop(self):
	85	def on_stop(self) -> None:
87	86	if self.timed_call:
88	87	try:
89	88	self.timed_call.cancel()

91	90	pass
92	91	self.timed_call = None
93	92
94		def on_new_notifications(self, max_token: RoomStreamToken):
95		# We just use the minimum stream ordering and ignore the vector clock
96		# component. This is safe to do as long as we always ignore the vector
97		# clock components.
98		max_stream_ordering = max_token.stream
99
100		if self.max_stream_ordering:
101		self.max_stream_ordering = max(
102		max_stream_ordering, self.max_stream_ordering
103		)
104		else:
105		self.max_stream_ordering = max_stream_ordering
106		self._start_processing()
107
108		def on_new_receipts(self, min_stream_id, max_stream_id):
	93	def on_new_receipts(self, min_stream_id: int, max_stream_id: int) -> None:
109	94	# We could wake up and cancel the timer but there tend to be quite a
110	95	# lot of read receipts so it's probably less work to just let the
111	96	# timer fire
112	97	pass
113	98
114		def on_timer(self):
	99	def on_timer(self) -> None:
115	100	self.timed_call = None
116	101	self._start_processing()
117	102
118		def _start_processing(self):
	103	def _start_processing(self) -> None:
119	104	if self._is_processing:
120	105	return
121	106
122	107	run_as_background_process("emailpush.process", self._process)
123	108
124		def _pause_processing(self):
	109	def _pause_processing(self) -> None:
125	110	"""Used by tests to temporarily pause processing of events.
126	111
127	112	Asserts that its not currently processing.

129	114	assert not self._is_processing
130	115	self._is_processing = True
131	116
132		def _resume_processing(self):
	117	def _resume_processing(self) -> None:
133	118	"""Used by tests to resume processing of events after pausing.
134	119	"""
135	120	assert self._is_processing
136	121	self._is_processing = False
137	122	self._start_processing()
138	123
139		async def _process(self):
	124	async def _process(self) -> None:
140	125	# we should never get here if we are already processing
141	126	assert not self._is_processing
142	127
143	128	try:
144	129	self._is_processing = True
145	130
146		if self.throttle_params is None:
	131	if not self._inited:
147	132	# this is our first loop: load up the throttle params
	133	assert self.pusher_id is not None
148	134	self.throttle_params = await self.store.get_throttle_params_by_room(
149	135	self.pusher_id
150	136	)
	137	self._inited = True
151	138
152	139	# if the max ordering changes while we're running _unsafe_process,
153	140	# call it again, and so on until we've caught up.

162	149	finally:
163	150	self._is_processing = False
164	151
165		async def _unsafe_process(self):
	152	async def _unsafe_process(self) -> None:
166	153	"""
167	154	Main logic of the push loop without the wrapper function that sets
168	155	up logging, measures and guards against multiple instances of it
169	156	being run.
170	157	"""
171	158	start = 0 if INCLUDE_ALL_UNREAD_NOTIFS else self.last_stream_ordering
172		fn = self.store.get_unread_push_actions_for_user_in_range_for_email
173		unprocessed = await fn(self.user_id, start, self.max_stream_ordering)
174
175		soonest_due_at = None
	159	unprocessed = await self.store.get_unread_push_actions_for_user_in_range_for_email(
	160	self.user_id, start, self.max_stream_ordering
	161	)
	162
	163	soonest_due_at = None # type: Optional[int]
176	164
177	165	if not unprocessed:
178	166	await self.save_last_stream_ordering_and_success(self.max_stream_ordering)

229	217	self.seconds_until(soonest_due_at), self.on_timer
230	218	)
231	219
232		async def save_last_stream_ordering_and_success(self, last_stream_ordering):
233		if last_stream_ordering is None:
234		# This happens if we haven't yet processed anything
235		return
236
	220	async def save_last_stream_ordering_and_success(
	221	self, last_stream_ordering: int
	222	) -> None:
237	223	self.last_stream_ordering = last_stream_ordering
238	224	pusher_still_exists = await self.store.update_pusher_last_stream_ordering_and_success(
239	225	self.app_id,

247	233	# lets just stop and return.
248	234	self.on_stop()
249	235
250		def seconds_until(self, ts_msec):
	236	def seconds_until(self, ts_msec: int) -> float:
251	237	secs = (ts_msec - self.clock.time_msec()) / 1000
252	238	return max(secs, 0)
253	239
254		def get_room_throttle_ms(self, room_id):
	240	def get_room_throttle_ms(self, room_id: str) -> int:
255	241	if room_id in self.throttle_params:
256		return self.throttle_params[room_id]["throttle_ms"]
	242	return self.throttle_params[room_id].throttle_ms
257	243	else:
258	244	return 0
259	245
260		def get_room_last_sent_ts(self, room_id):
	246	def get_room_last_sent_ts(self, room_id: str) -> int:
261	247	if room_id in self.throttle_params:
262		return self.throttle_params[room_id]["last_sent_ts"]
	248	return self.throttle_params[room_id].last_sent_ts
263	249	else:
264	250	return 0
265	251
266		def room_ready_to_notify_at(self, room_id):
	252	def room_ready_to_notify_at(self, room_id: str) -> int:
267	253	"""
268	254	Determines whether throttling should prevent us from sending an email
269	255	for the given room
270		Returns: The timestamp when we are next allowed to send an email notif
271		for this room
	256
	257	Returns:
	258	The timestamp when we are next allowed to send an email notif
	259	for this room
272	260	"""
273	261	last_sent_ts = self.get_room_last_sent_ts(room_id)
274	262	throttle_ms = self.get_room_throttle_ms(room_id)

276	264	may_send_at = last_sent_ts + throttle_ms
277	265	return may_send_at
278	266
279		async def sent_notif_update_throttle(self, room_id, notified_push_action):
	267	async def sent_notif_update_throttle(
	268	self, room_id: str, notified_push_action: dict
	269	) -> None:
280	270	# We have sent a notification, so update the throttle accordingly.
281	271	# If the event that triggered the notif happened more than
282	272	# THROTTLE_RESET_AFTER_MS after the previous one that triggered a

306	296	new_throttle_ms = min(
307	297	current_throttle_ms * THROTTLE_MULTIPLIER, THROTTLE_MAX_MS
308	298	)
309		self.throttle_params[room_id] = {
310		"last_sent_ts": self.clock.time_msec(),
311		"throttle_ms": new_throttle_ms,
312		}
	299	self.throttle_params[room_id] = ThrottleParams(
	300	self.clock.time_msec(), new_throttle_ms,
	301	)
	302	assert self.pusher_id is not None
313	303	await self.store.set_throttle_params(
314	304	self.pusher_id, room_id, self.throttle_params[room_id]
315	305	)
316	306
317		async def send_notification(self, push_actions, reason):
	307	async def send_notification(self, push_actions: List[dict], reason: dict) -> None:
318	308	logger.info("Sending notif email for user %r", self.user_id)
319	309
320	310	await self.mailer.send_notification_mail(

+57

-70

synapse/push/httppusher.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15	import logging
	16	import urllib.parse
	17	from typing import TYPE_CHECKING, Any, Dict, Iterable, Union
16	18
17	19	from prometheus_client import Counter
18	20
19	21	from twisted.internet.error import AlreadyCalled, AlreadyCancelled
20	22
21	23	from synapse.api.constants import EventTypes
	24	from synapse.events import EventBase
22	25	from synapse.logging import opentracing
23	26	from synapse.metrics.background_process_metrics import run_as_background_process
24		from synapse.push import PusherConfigException
25		from synapse.types import RoomStreamToken
	27	from synapse.push import Pusher, PusherConfig, PusherConfigException
26	28
27	29	from . import push_rule_evaluator, push_tools
	30
	31	if TYPE_CHECKING:
	32	from synapse.app.homeserver import HomeServer
28	33
29	34	logger = logging.getLogger(__name__)
30	35

49	54	)
50	55
51	56
52		class HttpPusher:
	57	class HttpPusher(Pusher):
53	58	INITIAL_BACKOFF_SEC = 1 # in seconds because that's what Twisted takes
54	59	MAX_BACKOFF_SEC = 60 * 60
55	60
56	61	# This one's in ms because we compare it against the clock
57	62	GIVE_UP_AFTER_MS = 24 * 60 * 60 * 1000
58	63
59		def __init__(self, hs, pusherdict):
60		self.hs = hs
61		self.store = self.hs.get_datastore()
	64	def __init__(self, hs: "HomeServer", pusher_config: PusherConfig):
	65	super().__init__(hs, pusher_config)
62	66	self.storage = self.hs.get_storage()
63		self.clock = self.hs.get_clock()
64		self.state_handler = self.hs.get_state_handler()
65		self.user_id = pusherdict["user_name"]
66		self.app_id = pusherdict["app_id"]
67		self.app_display_name = pusherdict["app_display_name"]
68		self.device_display_name = pusherdict["device_display_name"]
69		self.pushkey = pusherdict["pushkey"]
70		self.pushkey_ts = pusherdict["ts"]
71		self.data = pusherdict["data"]
72		self.last_stream_ordering = pusherdict["last_stream_ordering"]
	67	self.app_display_name = pusher_config.app_display_name
	68	self.device_display_name = pusher_config.device_display_name
	69	self.pushkey_ts = pusher_config.ts
	70	self.data = pusher_config.data
73	71	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
74		self.failing_since = pusherdict["failing_since"]
	72	self.failing_since = pusher_config.failing_since
75	73	self.timed_call = None
76	74	self._is_processing = False
77	75	self._group_unread_count_by_room = hs.config.push_group_unread_count_by_room
78	76
79		# This is the highest stream ordering we know it's safe to process.
80		# When new events arrive, we'll be given a window of new events: we
81		# should honour this rather than just looking for anything higher
82		# because of potential out-of-order event serialisation. This starts
83		# off as None though as we don't know any better.
84		self.max_stream_ordering = None
85
86		if "data" not in pusherdict:
87		raise PusherConfigException("No 'data' key for HTTP pusher")
88		self.data = pusherdict["data"]
	77	self.data = pusher_config.data
	78	if self.data is None:
	79	raise PusherConfigException("'data' key can not be null for HTTP pusher")
89	80
90	81	self.name = "%s/%s/%s" % (
91		pusherdict["user_name"],
92		pusherdict["app_id"],
93		pusherdict["pushkey"],
	82	pusher_config.user_name,
	83	pusher_config.app_id,
	84	pusher_config.pushkey,
94	85	)
95	86
96		if self.data is None:
97		raise PusherConfigException("data can not be null for HTTP pusher")
98
	87	# Validate that there's a URL and it is of the proper form.
99	88	if "url" not in self.data:
100	89	raise PusherConfigException("'url' required in data for HTTP pusher")
101		self.url = self.data["url"]
102		self.http_client = hs.get_proxied_http_client()
	90
	91	url = self.data["url"]
	92	if not isinstance(url, str):
	93	raise PusherConfigException("'url' must be a string")
	94	url_parts = urllib.parse.urlparse(url)
	95	# Note that the specification also says the scheme must be HTTPS, but
	96	# it isn't up to the homeserver to verify that.
	97	if url_parts.path != "/_matrix/push/v1/notify":
	98	raise PusherConfigException(
	99	"'url' must have a path of '/_matrix/push/v1/notify'"
	100	)
	101
	102	self.url = url
	103	self.http_client = hs.get_proxied_blacklisted_http_client()
103	104	self.data_minus_url = {}
104	105	self.data_minus_url.update(self.data)
105	106	del self.data_minus_url["url"]
106	107
107		def on_started(self, should_check_for_notifs):
	108	def on_started(self, should_check_for_notifs: bool) -> None:
108	109	"""Called when this pusher has been started.
109	110
110	111	Args:
111		should_check_for_notifs (bool): Whether we should immediately
	112	should_check_for_notifs: Whether we should immediately
112	113	check for push to send. Set to False only if it's known there
113	114	is nothing to send
114	115	"""
115	116	if should_check_for_notifs:
116	117	self._start_processing()
117	118
118		def on_new_notifications(self, max_token: RoomStreamToken):
119		# We just use the minimum stream ordering and ignore the vector clock
120		# component. This is safe to do as long as we always ignore the vector
121		# clock components.
122		max_stream_ordering = max_token.stream
123
124		self.max_stream_ordering = max(
125		max_stream_ordering, self.max_stream_ordering or 0
126		)
127		self._start_processing()
128
129		def on_new_receipts(self, min_stream_id, max_stream_id):
	119	def on_new_receipts(self, min_stream_id: int, max_stream_id: int) -> None:
130	120	# Note that the min here shouldn't be relied upon to be accurate.
131	121
132	122	# We could check the receipts are actually m.read receipts here,
133	123	# but currently that's the only type of receipt anyway...
134	124	run_as_background_process("http_pusher.on_new_receipts", self._update_badge)
135	125
136		async def _update_badge(self):
	126	async def _update_badge(self) -> None:
137	127	# XXX as per https://github.com/matrix-org/matrix-doc/issues/2627, this seems
138	128	# to be largely redundant. perhaps we can remove it.
139	129	badge = await push_tools.get_badge_count(

143	133	)
144	134	await self._send_badge(badge)
145	135
146		def on_timer(self):
	136	def on_timer(self) -> None:
147	137	self._start_processing()
148	138
149		def on_stop(self):
	139	def on_stop(self) -> None:
150	140	if self.timed_call:
151	141	try:
152	142	self.timed_call.cancel()

154	144	pass
155	145	self.timed_call = None
156	146
157		def _start_processing(self):
	147	def _start_processing(self) -> None:
158	148	if self._is_processing:
159	149	return
160	150
161	151	run_as_background_process("httppush.process", self._process)
162	152
163		async def _process(self):
	153	async def _process(self) -> None:
164	154	# we should never get here if we are already processing
165	155	assert not self._is_processing
166	156

179	169	finally:
180	170	self._is_processing = False
181	171
182		async def _unsafe_process(self):
	172	async def _unsafe_process(self) -> None:
183	173	"""
184	174	Looks for unset notifications and dispatch them, in order
185	175	Never call this directly: use _process which will only allow this to
186	176	run once per pusher.
187	177	"""
188
189		fn = self.store.get_unread_push_actions_for_user_in_range_for_http
190		unprocessed = await fn(
	178	unprocessed = await self.store.get_unread_push_actions_for_user_in_range_for_http(
191	179	self.user_id, self.last_stream_ordering, self.max_stream_ordering
192	180	)
193	181

256	244	)
257	245	self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
258	246	self.last_stream_ordering = push_action["stream_ordering"]
259		pusher_still_exists = await self.store.update_pusher_last_stream_ordering(
	247	await self.store.update_pusher_last_stream_ordering(
260	248	self.app_id,
261	249	self.pushkey,
262	250	self.user_id,
263	251	self.last_stream_ordering,
264	252	)
265		if not pusher_still_exists:
266		# The pusher has been deleted while we were processing, so
267		# lets just stop and return.
268		self.on_stop()
269		return
270	253
271	254	self.failing_since = None
272	255	await self.store.update_pusher_failing_since(

282	265	)
283	266	break
284	267
285		async def _process_one(self, push_action):
	268	async def _process_one(self, push_action: dict) -> bool:
286	269	if "notify" not in push_action["actions"]:
287	270	return True
288	271

313	296	await self.hs.remove_pusher(self.app_id, pk, self.user_id)
314	297	return True
315	298
316		async def _build_notification_dict(self, event, tweaks, badge):
	299	async def _build_notification_dict(
	300	self, event: EventBase, tweaks: Dict[str, bool], badge: int
	301	) -> Dict[str, Any]:
317	302	priority = "low"
318	303	if (
319	304	event.type == EventTypes.Encrypted

324	309	# or may do so (i.e. is encrypted so has unknown effects).
325	310	priority = "high"
326	311
	312	# This was checked in the __init__, but mypy doesn't seem to know that.
	313	assert self.data is not None
327	314	if self.data.get("format") == "event_id_only":
328	315	d = {
329	316	"notification": {

343	330	}
344	331	return d
345	332
346		ctx = await push_tools.get_context_for_event(
347		self.storage, self.state_handler, event, self.user_id
348		)
	333	ctx = await push_tools.get_context_for_event(self.storage, event, self.user_id)
349	334
350	335	d = {
351	336	"notification": {

385	370
386	371	return d
387	372
388		async def dispatch_push(self, event, tweaks, badge):
	373	async def dispatch_push(
	374	self, event: EventBase, tweaks: Dict[str, bool], badge: int
	375	) -> Union[bool, Iterable[str]]:
389	376	notification_dict = await self._build_notification_dict(event, tweaks, badge)
390	377	if not notification_dict:
391	378	return []

+90

-43

synapse/push/mailer.py less more

18	18	import urllib.parse
19	19	from email.mime.multipart import MIMEMultipart
20	20	from email.mime.text import MIMEText
21		from typing import Iterable, List, TypeVar
	21	from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, TypeVar
22	22
23	23	import bleach
24	24	import jinja2

26	26	from synapse.api.constants import EventTypes, Membership
27	27	from synapse.api.errors import StoreError
28	28	from synapse.config.emailconfig import EmailSubjectConfig
	29	from synapse.events import EventBase
29	30	from synapse.logging.context import make_deferred_yieldable
30	31	from synapse.push.presentable_names import (
31	32	calculate_room_name,
32	33	descriptor_from_member_events,
33	34	name_from_member_event,
34	35	)
35		from synapse.types import UserID
	36	from synapse.types import StateMap, UserID
36	37	from synapse.util.async_helpers import concurrently_execute
37	38	from synapse.visibility import filter_events_for_client
	39
	40	if TYPE_CHECKING:
	41	from synapse.app.homeserver import HomeServer
38	42
39	43	logger = logging.getLogger(__name__)
40	44

92	96
93	97
94	98	class Mailer:
95		def __init__(self, hs, app_name, template_html, template_text):
	99	def __init__(
	100	self,
	101	hs: "HomeServer",
	102	app_name: str,
	103	template_html: jinja2.Template,
	104	template_text: jinja2.Template,
	105	):
96	106	self.hs = hs
97	107	self.template_html = template_html
98	108	self.template_text = template_text

107	117
108	118	logger.info("Created Mailer for app_name %s" % app_name)
109	119
110		async def send_password_reset_mail(self, email_address, token, client_secret, sid):
	120	async def send_password_reset_mail(
	121	self, email_address: str, token: str, client_secret: str, sid: str
	122	) -> None:
111	123	"""Send an email with a password reset link to a user
112	124
113	125	Args:
114		email_address (str): Email address we're sending the password
	126	email_address: Email address we're sending the password
115	127	reset to
116		token (str): Unique token generated by the server to verify
	128	token: Unique token generated by the server to verify
117	129	the email was received
118		client_secret (str): Unique token generated by the client to
	130	client_secret: Unique token generated by the client to
119	131	group together multiple email sending attempts
120		sid (str): The generated session ID
	132	sid: The generated session ID
121	133	"""
122	134	params = {"token": token, "client_secret": client_secret, "sid": sid}
123	135	link = (

135	147	template_vars,
136	148	)
137	149
138		async def send_registration_mail(self, email_address, token, client_secret, sid):
	150	async def send_registration_mail(
	151	self, email_address: str, token: str, client_secret: str, sid: str
	152	) -> None:
139	153	"""Send an email with a registration confirmation link to a user
140	154
141	155	Args:
142		email_address (str): Email address we're sending the registration
	156	email_address: Email address we're sending the registration
143	157	link to
144		token (str): Unique token generated by the server to verify
	158	token: Unique token generated by the server to verify
145	159	the email was received
146		client_secret (str): Unique token generated by the client to
	160	client_secret: Unique token generated by the client to
147	161	group together multiple email sending attempts
148		sid (str): The generated session ID
	162	sid: The generated session ID
149	163	"""
150	164	params = {"token": token, "client_secret": client_secret, "sid": sid}
151	165	link = (

163	177	template_vars,
164	178	)
165	179
166		async def send_add_threepid_mail(self, email_address, token, client_secret, sid):
	180	async def send_add_threepid_mail(
	181	self, email_address: str, token: str, client_secret: str, sid: str
	182	) -> None:
167	183	"""Send an email with a validation link to a user for adding a 3pid to their account
168	184
169	185	Args:
170		email_address (str): Email address we're sending the validation link to
171
172		token (str): Unique token generated by the server to verify the email was received
173
174		client_secret (str): Unique token generated by the client to group together
	186	email_address: Email address we're sending the validation link to
	187
	188	token: Unique token generated by the server to verify the email was received
	189
	190	client_secret: Unique token generated by the client to group together
175	191	multiple email sending attempts
176	192
177		sid (str): The generated session ID
	193	sid: The generated session ID
178	194	"""
179	195	params = {"token": token, "client_secret": client_secret, "sid": sid}
180	196	link = (

193	209	)
194	210
195	211	async def send_notification_mail(
196		self, app_id, user_id, email_address, push_actions, reason
197		):
	212	self,
	213	app_id: str,
	214	user_id: str,
	215	email_address: str,
	216	push_actions: Iterable[Dict[str, Any]],
	217	reason: Dict[str, Any],
	218	) -> None:
198	219	"""Send email regarding a user's room notifications"""
199	220	rooms_in_order = deduped_ordered_list([pa["room_id"] for pa in push_actions])
200	221

202	223	[pa["event_id"] for pa in push_actions]
203	224	)
204	225
205		notifs_by_room = {}
	226	notifs_by_room = {} # type: Dict[str, List[Dict[str, Any]]]
206	227	for pa in push_actions:
207	228	notifs_by_room.setdefault(pa["room_id"], []).append(pa)
208	229

261	282
262	283	await self.send_email(email_address, summary_text, template_vars)
263	284
264		async def send_email(self, email_address, subject, extra_template_vars):
	285	async def send_email(
	286	self, email_address: str, subject: str, extra_template_vars: Dict[str, Any]
	287	) -> None:
265	288	"""Send an email with the given information and template text"""
266	289	try:
267	290	from_string = self.hs.config.email_notif_from % {"app": self.app_name}

314	337	)
315	338
316	339	async def get_room_vars(
317		self, room_id, user_id, notifs, notif_events, room_state_ids
318		):
	340	self,
	341	room_id: str,
	342	user_id: str,
	343	notifs: Iterable[Dict[str, Any]],
	344	notif_events: Dict[str, EventBase],
	345	room_state_ids: StateMap[str],
	346	) -> Dict[str, Any]:
319	347	# Check if one of the notifs is an invite event for the user.
320	348	is_invite = False
321	349	for n in notifs:

333	361	"notifs": [],
334	362	"invite": is_invite,
335	363	"link": self.make_room_link(room_id),
336		}
	364	} # type: Dict[str, Any]
337	365
338	366	if not is_invite:
339	367	for n in notifs:

364	392
365	393	return room_vars
366	394
367		async def get_notif_vars(self, notif, user_id, notif_event, room_state_ids):
	395	async def get_notif_vars(
	396	self,
	397	notif: Dict[str, Any],
	398	user_id: str,
	399	notif_event: EventBase,
	400	room_state_ids: StateMap[str],
	401	) -> Dict[str, Any]:
368	402	results = await self.store.get_events_around(
369	403	notif["room_id"],
370	404	notif["event_id"],

390	424
391	425	return ret
392	426
393		async def get_message_vars(self, notif, event, room_state_ids):
	427	async def get_message_vars(
	428	self, notif: Dict[str, Any], event: EventBase, room_state_ids: StateMap[str]
	429	) -> Optional[Dict[str, Any]]:
394	430	if event.type != EventTypes.Message and event.type != EventTypes.Encrypted:
395	431	return None
396	432

431	467
432	468	return ret
433	469
434		def add_text_message_vars(self, messagevars, event):
	470	def add_text_message_vars(
	471	self, messagevars: Dict[str, Any], event: EventBase
	472	) -> None:
435	473	msgformat = event.content.get("format")
436	474
437	475	messagevars["format"] = msgformat

444	482	elif body:
445	483	messagevars["body_text_html"] = safe_text(body)
446	484
447		return messagevars
448
449		def add_image_message_vars(self, messagevars, event):
450		messagevars["image_url"] = event.content["url"]
451
452		return messagevars
	485	def add_image_message_vars(
	486	self, messagevars: Dict[str, Any], event: EventBase
	487	) -> None:
	488	"""
	489	Potentially add an image URL to the message variables.
	490	"""
	491	if "url" in event.content:
	492	messagevars["image_url"] = event.content["url"]
453	493
454	494	async def make_summary_text(
455		self, notifs_by_room, room_state_ids, notif_events, user_id, reason
	495	self,
	496	notifs_by_room: Dict[str, List[Dict[str, Any]]],
	497	room_state_ids: Dict[str, StateMap[str]],
	498	notif_events: Dict[str, EventBase],
	499	user_id: str,
	500	reason: Dict[str, Any],
456	501	):
457	502	if len(notifs_by_room) == 1:
458	503	# Only one room has new stuff

579	624	"app": self.app_name,
580	625	}
581	626
582		def make_room_link(self, room_id):
	627	def make_room_link(self, room_id: str) -> str:
583	628	if self.hs.config.email_riot_base_url:
584	629	base_url = "%s/#/room" % (self.hs.config.email_riot_base_url)
585	630	elif self.app_name == "Vector":

589	634	base_url = "https://matrix.to/#"
590	635	return "%s/%s" % (base_url, room_id)
591	636
592		def make_notif_link(self, notif):
	637	def make_notif_link(self, notif: Dict[str, str]) -> str:
593	638	if self.hs.config.email_riot_base_url:
594	639	return "%s/#/room/%s/%s" % (
595	640	self.hs.config.email_riot_base_url,

605	650	else:
606	651	return "https://matrix.to/#/%s/%s" % (notif["room_id"], notif["event_id"])
607	652
608		def make_unsubscribe_link(self, user_id, app_id, email_address):
	653	def make_unsubscribe_link(
	654	self, user_id: str, app_id: str, email_address: str
	655	) -> str:
609	656	params = {
610	657	"access_token": self.macaroon_gen.generate_delete_pusher_token(user_id),
611	658	"app_id": app_id,

619	666	)
620	667
621	668
622		def safe_markup(raw_html):
	669	def safe_markup(raw_html: str) -> jinja2.Markup:
623	670	return jinja2.Markup(
624	671	bleach.linkify(
625	672	bleach.clean(

634	681	)
635	682
636	683
637		def safe_text(raw_text):
	684	def safe_text(raw_text: str) -> jinja2.Markup:
638	685	"""
639	686	Process text: treat it as HTML but escape any tags (ie. just escape the
640	687	HTML) then linkify it.

654	701	return ret
655	702
656	703
657		def string_ordinal_total(s):
	704	def string_ordinal_total(s: str) -> int:
658	705	tot = 0
659	706	for c in s:
660	707	tot += ord(c)

+31

-21

synapse/push/presentable_names.py less more

14	14
15	15	import logging
16	16	import re
	17	from typing import TYPE_CHECKING, Dict, Iterable, Optional
17	18
18	19	from synapse.api.constants import EventTypes
	20	from synapse.events import EventBase
	21	from synapse.types import StateMap
	22
	23	if TYPE_CHECKING:
	24	from synapse.storage.databases.main import DataStore
19	25
20	26	logger = logging.getLogger(__name__)
21	27

27	33
28	34
29	35	async def calculate_room_name(
30		store,
31		room_state_ids,
32		user_id,
33		fallback_to_members=True,
34		fallback_to_single_member=True,
35		):
	36	store: "DataStore",
	37	room_state_ids: StateMap[str],
	38	user_id: str,
	39	fallback_to_members: bool = True,
	40	fallback_to_single_member: bool = True,
	41	) -> Optional[str]:
36	42	"""
37	43	Works out a user-facing name for the given room as per Matrix
38	44	spec recommendations.
39	45	Does not yet support internationalisation.
40	46	Args:
41		room_state: Dictionary of the room's state
	47	store: The data store to query.
	48	room_state_ids: Dictionary of the room's state IDs.
42	49	user_id: The ID of the user to whom the room name is being presented
43	50	fallback_to_members: If False, return None instead of generating a name
44	51	based on the room's members if the room has no
45	52	title or aliases.
	53	fallback_to_single_member: If False, return None instead of generating a
	54	name based on the user who invited this user to the room if the room
	55	has no title or aliases.
46	56
47	57	Returns:
48		(string or None) A human readable name for the room.
	58	A human readable name for the room, if possible.
49	59	"""
50	60	# does it have a name?
51	61	if (EventTypes.Name, "") in room_state_ids:

96	106	name_from_member_event(inviter_member_event),
97	107	)
98	108	else:
99		return
	109	return None
100	110	else:
101	111	return "Room Invite"
102	112

149	159	else:
150	160	return ALL_ALONE
151	161	elif len(other_members) == 1 and not fallback_to_single_member:
152		return
153		else:
154		return descriptor_from_member_events(other_members)
155
156
157		def descriptor_from_member_events(member_events):
	162	return None
	163
	164	return descriptor_from_member_events(other_members)
	165
	166
	167	def descriptor_from_member_events(member_events: Iterable[EventBase]) -> str:
158	168	"""Get a description of the room based on the member events.
159	169
160	170	Args:
161		member_events (Iterable[FrozenEvent])
	171	member_events: The events of a room.
162	172
163	173	Returns:
164		str
	174	The room description
165	175	"""
166	176
167	177	member_events = list(member_events)

182	192	)
183	193
184	194
185		def name_from_member_event(member_event):
	195	def name_from_member_event(member_event: EventBase) -> str:
186	196	if (
187	197	member_event.content
188	198	and "displayname" in member_event.content

192	202	return member_event.state_key
193	203
194	204
195		def _state_as_two_level_dict(state):
196		ret = {}
	205	def _state_as_two_level_dict(state: StateMap[str]) -> Dict[str, Dict[str, str]]:
	206	ret = {} # type: Dict[str, Dict[str, str]]
197	207	for k, v in state.items():
198	208	ret.setdefault(k[0], {})[k[1]] = v
199	209	return ret
200	210
201	211
202		def _looks_like_an_alias(string):
	212	def _looks_like_an_alias(string: str) -> bool:
203	213	return ALIAS_RE.match(string) is not None

+22

-6

synapse/push/push_rule_evaluator.py less more

29	29	INEQUALITY_EXPR = re.compile("^([=<>])([0-9])$")
30	30
31	31
32		def _room_member_count(ev, condition, room_member_count):
	32	def _room_member_count(
	33	ev: EventBase, condition: Dict[str, Any], room_member_count: int
	34	) -> bool:
33	35	return _test_ineq_condition(condition, room_member_count)
34	36
35	37
36		def _sender_notification_permission(ev, condition, sender_power_level, power_levels):
	38	def _sender_notification_permission(
	39	ev: EventBase,
	40	condition: Dict[str, Any],
	41	sender_power_level: int,
	42	power_levels: Dict[str, Union[int, Dict[str, int]]],
	43	) -> bool:
37	44	notif_level_key = condition.get("key")
38	45	if notif_level_key is None:
39	46	return False
40	47
41	48	notif_levels = power_levels.get("notifications", {})
	49	assert isinstance(notif_levels, dict)
42	50	room_notif_level = notif_levels.get(notif_level_key, 50)
43	51
44	52	return sender_power_level >= room_notif_level
45	53
46	54
47		def _test_ineq_condition(condition, number):
	55	def _test_ineq_condition(condition: Dict[str, Any], number: int) -> bool:
48	56	if "is" not in condition:
49	57	return False
50	58	m = INEQUALITY_EXPR.match(condition["is"])

109	117	event: EventBase,
110	118	room_member_count: int,
111	119	sender_power_level: int,
112		power_levels: dict,
	120	power_levels: Dict[str, Union[int, Dict[str, int]]],
113	121	):
114	122	self._event = event
115	123	self._room_member_count = room_member_count

119	127	# Maps strings of e.g. 'content.body' -> event["content"]["body"]
120	128	self._value_cache = _flatten_dict(event)
121	129
122		def matches(self, condition: dict, user_id: str, display_name: str) -> bool:
	130	def matches(
	131	self, condition: Dict[str, Any], user_id: str, display_name: str
	132	) -> bool:
123	133	if condition["kind"] == "event_match":
124	134	return self._event_match(condition, user_id)
125	135	elif condition["kind"] == "contains_display_name":

260	270	return r"(^\|\W)%s(\W\|$)" % (r,)
261	271
262	272
263		def _flatten_dict(d, prefix=[], result=None):
	273	def _flatten_dict(
	274	d: Union[EventBase, dict],
	275	prefix: Optional[List[str]] = None,
	276	result: Optional[Dict[str, str]] = None,
	277	) -> Dict[str, str]:
	278	if prefix is None:
	279	prefix = []
264	280	if result is None:
265	281	result = {}
266	282	for key, value in d.items():

+6

-1

synapse/push/push_tools.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14	from typing import Dict
	15
	16	from synapse.events import EventBase
14	17	from synapse.push.presentable_names import calculate_room_name, name_from_member_event
15	18	from synapse.storage import Storage
16	19	from synapse.storage.databases.main import DataStore

45	48	return badge
46	49
47	50
48		async def get_context_for_event(storage: Storage, state_handler, ev, user_id):
	51	async def get_context_for_event(
	52	storage: Storage, ev: EventBase, user_id: str
	53	) -> Dict[str, str]:
49	54	ctx = {}
50	55
51	56	room_state_ids = await storage.state.get_state_ids_for_event(ev.event_id)

+21

-13

synapse/push/pusher.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import TYPE_CHECKING, Callable, Dict, Optional
16	17
	18	from synapse.push import Pusher, PusherConfig
17	19	from synapse.push.emailpusher import EmailPusher
	20	from synapse.push.httppusher import HttpPusher
18	21	from synapse.push.mailer import Mailer
19	22
20		from .httppusher import HttpPusher
	23	if TYPE_CHECKING:
	24	from synapse.app.homeserver import HomeServer
21	25
22	26	logger = logging.getLogger(__name__)
23	27
24	28
25	29	class PusherFactory:
26		def __init__(self, hs):
	30	def __init__(self, hs: "HomeServer"):
27	31	self.hs = hs
28	32	self.config = hs.config
29	33
30		self.pusher_types = {"http": HttpPusher}
	34	self.pusher_types = {
	35	"http": HttpPusher
	36	} # type: Dict[str, Callable[[HomeServer, PusherConfig], Pusher]]
31	37
32	38	logger.info("email enable notifs: %r", hs.config.email_enable_notifs)
33	39	if hs.config.email_enable_notifs:
34		self.mailers = {} # app_name -> Mailer
	40	self.mailers = {} # type: Dict[str, Mailer]
35	41
36	42	self._notif_template_html = hs.config.email_notif_template_html
37	43	self._notif_template_text = hs.config.email_notif_template_text

40	46
41	47	logger.info("defined email pusher type")
42	48
43		def create_pusher(self, pusherdict):
44		kind = pusherdict["kind"]
	49	def create_pusher(self, pusher_config: PusherConfig) -> Optional[Pusher]:
	50	kind = pusher_config.kind
45	51	f = self.pusher_types.get(kind, None)
46	52	if not f:
47	53	return None
48		logger.debug("creating %s pusher for %r", kind, pusherdict)
49		return f(self.hs, pusherdict)
	54	logger.debug("creating %s pusher for %r", kind, pusher_config)
	55	return f(self.hs, pusher_config)
50	56
51		def _create_email_pusher(self, _hs, pusherdict):
52		app_name = self._app_name_from_pusherdict(pusherdict)
	57	def _create_email_pusher(
	58	self, _hs: "HomeServer", pusher_config: PusherConfig
	59	) -> EmailPusher:
	60	app_name = self._app_name_from_pusherdict(pusher_config)
53	61	mailer = self.mailers.get(app_name)
54	62	if not mailer:
55	63	mailer = Mailer(

59	67	template_text=self._notif_template_text,
60	68	)
61	69	self.mailers[app_name] = mailer
62		return EmailPusher(self.hs, pusherdict, mailer)
	70	return EmailPusher(self.hs, pusher_config, mailer)
63	71
64		def _app_name_from_pusherdict(self, pusherdict):
65		data = pusherdict["data"]
	72	def _app_name_from_pusherdict(self, pusher_config: PusherConfig) -> str:
	73	data = pusher_config.data
66	74
67	75	if isinstance(data, dict):
68	76	brand = data.get("brand")

+94

-90

synapse/push/pusherpool.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
17		from typing import TYPE_CHECKING, Dict, Union
	17	from typing import TYPE_CHECKING, Dict, Iterable, Optional
18	18
19	19	from prometheus_client import Gauge
20	20

22	22	run_as_background_process,
23	23	wrap_as_background_process,
24	24	)
25		from synapse.push import PusherConfigException
26		from synapse.push.emailpusher import EmailPusher
27		from synapse.push.httppusher import HttpPusher
	25	from synapse.push import Pusher, PusherConfig, PusherConfigException
28	26	from synapse.push.pusher import PusherFactory
29		from synapse.types import RoomStreamToken
	27	from synapse.types import JsonDict, RoomStreamToken
30	28	from synapse.util.async_helpers import concurrently_execute
31	29
32	30	if TYPE_CHECKING:

76	74	self._last_room_stream_id_seen = self.store.get_room_max_stream_ordering()
77	75
78	76	# map from user id to app_id:pushkey to pusher
79		self.pushers = {} # type: Dict[str, Dict[str, Union[HttpPusher, EmailPusher]]]
80
81		def start(self):
	77	self.pushers = {} # type: Dict[str, Dict[str, Pusher]]
	78
	79	def start(self) -> None:
82	80	"""Starts the pushers off in a background process.
83	81	"""
84	82	if not self._should_start_pushers:

88	86
89	87	async def add_pusher(
90	88	self,
91		user_id,
92		access_token,
93		kind,
94		app_id,
95		app_display_name,
96		device_display_name,
97		pushkey,
98		lang,
99		data,
100		profile_tag="",
101		):
	89	user_id: str,
	90	access_token: Optional[int],
	91	kind: str,
	92	app_id: str,
	93	app_display_name: str,
	94	device_display_name: str,
	95	pushkey: str,
	96	lang: Optional[str],
	97	data: JsonDict,
	98	profile_tag: str = "",
	99	) -> Optional[Pusher]:
102	100	"""Creates a new pusher and adds it to the pool
103	101
104	102	Returns:
105		EmailPusher\|HttpPusher
	103	The newly created pusher.
106	104	"""
107	105
108	106	time_now_msec = self.clock.time_msec()
	107
	108	# create the pusher setting last_stream_ordering to the current maximum
	109	# stream ordering, so it will process pushes from this point onwards.
	110	last_stream_ordering = self.store.get_room_max_stream_ordering()
109	111
110	112	# we try to create the pusher just to validate the config: it
111	113	# will then get pulled out of the database,
112	114	# recreated, added and started: this means we have only one
113	115	# code path adding pushers.
114	116	self.pusher_factory.create_pusher(
115		{
116		"id": None,
117		"user_name": user_id,
118		"kind": kind,
119		"app_id": app_id,
120		"app_display_name": app_display_name,
121		"device_display_name": device_display_name,
122		"pushkey": pushkey,
123		"ts": time_now_msec,
124		"lang": lang,
125		"data": data,
126		"last_stream_ordering": None,
127		"last_success": None,
128		"failing_since": None,
129		}
	117	PusherConfig(
	118	id=None,
	119	user_name=user_id,
	120	access_token=access_token,
	121	profile_tag=profile_tag,
	122	kind=kind,
	123	app_id=app_id,
	124	app_display_name=app_display_name,
	125	device_display_name=device_display_name,
	126	pushkey=pushkey,
	127	ts=time_now_msec,
	128	lang=lang,
	129	data=data,
	130	last_stream_ordering=last_stream_ordering,
	131	last_success=None,
	132	failing_since=None,
	133	)
130	134	)
131
132		# create the pusher setting last_stream_ordering to the current maximum
133		# stream ordering in event_push_actions, so it will process
134		# pushes from this point onwards.
135		last_stream_ordering = await self.store.get_latest_push_action_stream_ordering()
136	135
137	136	await self.store.add_pusher(
138	137	user_id=user_id,

153	152	return pusher
154	153
155	154	async def remove_pushers_by_app_id_and_pushkey_not_user(
156		self, app_id, pushkey, not_user_id
157		):
	155	self, app_id: str, pushkey: str, not_user_id: str
	156	) -> None:
158	157	to_remove = await self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
159	158	for p in to_remove:
160		if p["user_name"] != not_user_id:
	159	if p.user_name != not_user_id:
161	160	logger.info(
162	161	"Removing pusher for app id %s, pushkey %s, user %s",
163	162	app_id,
164	163	pushkey,
165		p["user_name"],
	164	p.user_name,
166	165	)
167		await self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
168
169		async def remove_pushers_by_access_token(self, user_id, access_tokens):
	166	await self.remove_pusher(p.app_id, p.pushkey, p.user_name)
	167
	168	async def remove_pushers_by_access_token(
	169	self, user_id: str, access_tokens: Iterable[int]
	170	) -> None:
170	171	"""Remove the pushers for a given user corresponding to a set of
171	172	access_tokens.
172	173
173	174	Args:
174		user_id (str): user to remove pushers for
175		access_tokens (Iterable[int]): access token ids to remove pushers
176		for
	175	user_id: user to remove pushers for
	176	access_tokens: access token ids to remove pushers for
177	177	"""
178	178	if not self._pusher_shard_config.should_handle(self._instance_name, user_id):
179	179	return
180	180
181	181	tokens = set(access_tokens)
182	182	for p in await self.store.get_pushers_by_user_id(user_id):
183		if p["access_token"] in tokens:
	183	if p.access_token in tokens:
184	184	logger.info(
185	185	"Removing pusher for app id %s, pushkey %s, user %s",
186		p["app_id"],
187		p["pushkey"],
188		p["user_name"],
	186	p.app_id,
	187	p.pushkey,
	188	p.user_name,
189	189	)
190		await self.remove_pusher(p["app_id"], p["pushkey"], p["user_name"])
191
192		def on_new_notifications(self, max_token: RoomStreamToken):
	190	await self.remove_pusher(p.app_id, p.pushkey, p.user_name)
	191
	192	def on_new_notifications(self, max_token: RoomStreamToken) -> None:
193	193	if not self.pushers:
194	194	# nothing to do here.
195	195	return

208	208	self._on_new_notifications(max_token)
209	209
210	210	@wrap_as_background_process("on_new_notifications")
211		async def _on_new_notifications(self, max_token: RoomStreamToken):
	211	async def _on_new_notifications(self, max_token: RoomStreamToken) -> None:
212	212	# We just use the minimum stream ordering and ignore the vector clock
213	213	# component. This is safe to do as long as we always ignore the vector
214	214	# clock components.

238	238	except Exception:
239	239	logger.exception("Exception in pusher on_new_notifications")
240	240
241		async def on_new_receipts(self, min_stream_id, max_stream_id, affected_room_ids):
	241	async def on_new_receipts(
	242	self, min_stream_id: int, max_stream_id: int, affected_room_ids: Iterable[str]
	243	) -> None:
242	244	if not self.pushers:
243	245	# nothing to do here.
244	246	return

266	268	except Exception:
267	269	logger.exception("Exception in pusher on_new_receipts")
268	270
269		async def start_pusher_by_id(self, app_id, pushkey, user_id):
	271	async def start_pusher_by_id(
	272	self, app_id: str, pushkey: str, user_id: str
	273	) -> Optional[Pusher]:
270	274	"""Look up the details for the given pusher, and start it
271	275
272	276	Returns:
273		EmailPusher\|HttpPusher\|None: The pusher started, if any
	277	The pusher started, if any
274	278	"""
275	279	if not self._should_start_pushers:
276		return
	280	return None
277	281
278	282	if not self._pusher_shard_config.should_handle(self._instance_name, user_id):
279		return
	283	return None
280	284
281	285	resultlist = await self.store.get_pushers_by_app_id_and_pushkey(app_id, pushkey)
282	286
283		pusher_dict = None
	287	pusher_config = None
284	288	for r in resultlist:
285		if r["user_name"] == user_id:
286		pusher_dict = r
	289	if r.user_name == user_id:
	290	pusher_config = r
287	291
288	292	pusher = None
289		if pusher_dict:
290		pusher = await self._start_pusher(pusher_dict)
	293	if pusher_config:
	294	pusher = await self._start_pusher(pusher_config)
291	295
292	296	return pusher
293	297

302	306
303	307	logger.info("Started pushers")
304	308
305		async def _start_pusher(self, pusherdict):
	309	async def _start_pusher(self, pusher_config: PusherConfig) -> Optional[Pusher]:
306	310	"""Start the given pusher
307	311
308	312	Args:
309		pusherdict (dict): dict with the values pulled from the db table
	313	pusher_config: The pusher configuration with the values pulled from the db table
310	314
311	315	Returns:
312		EmailPusher\|HttpPusher
	316	The newly created pusher or None.
313	317	"""
314	318	if not self._pusher_shard_config.should_handle(
315		self._instance_name, pusherdict["user_name"]
	319	self._instance_name, pusher_config.user_name
316	320	):
317		return
	321	return None
318	322
319	323	try:
320		p = self.pusher_factory.create_pusher(pusherdict)
	324	p = self.pusher_factory.create_pusher(pusher_config)
321	325	except PusherConfigException as e:
322	326	logger.warning(
323	327	"Pusher incorrectly configured id=%i, user=%s, appid=%s, pushkey=%s: %s",
324		pusherdict["id"],
325		pusherdict.get("user_name"),
326		pusherdict.get("app_id"),
327		pusherdict.get("pushkey"),
	328	pusher_config.id,
	329	pusher_config.user_name,
	330	pusher_config.app_id,
	331	pusher_config.pushkey,
328	332	e,
329	333	)
330		return
	334	return None
331	335	except Exception:
332	336	logger.exception(
333		"Couldn't start pusher id %i: caught Exception", pusherdict["id"],
334		)
335		return
	337	"Couldn't start pusher id %i: caught Exception", pusher_config.id,
	338	)
	339	return None
336	340
337	341	if not p:
338		return
339
340		appid_pushkey = "%s:%s" % (pusherdict["app_id"], pusherdict["pushkey"])
341
342		byuser = self.pushers.setdefault(pusherdict["user_name"], {})
	342	return None
	343
	344	appid_pushkey = "%s:%s" % (pusher_config.app_id, pusher_config.pushkey)
	345
	346	byuser = self.pushers.setdefault(pusher_config.user_name, {})
343	347	if appid_pushkey in byuser:
344	348	byuser[appid_pushkey].on_stop()
345	349	byuser[appid_pushkey] = p

349	353	# Check if there may be push to process. We do this as this check is a
350	354	# lot cheaper to do than actually fetching the exact rows we need to
351	355	# push.
352		user_id = pusherdict["user_name"]
353		last_stream_ordering = pusherdict["last_stream_ordering"]
	356	user_id = pusher_config.user_name
	357	last_stream_ordering = pusher_config.last_stream_ordering
354	358	if last_stream_ordering:
355	359	have_notifs = await self.store.get_if_maybe_push_in_range_for_user(
356	360	user_id, last_stream_ordering

364	368
365	369	return p
366	370
367		async def remove_pusher(self, app_id, pushkey, user_id):
	371	async def remove_pusher(self, app_id: str, pushkey: str, user_id: str) -> None:
368	372	appid_pushkey = "%s:%s" % (app_id, pushkey)
369	373
370	374	byuser = self.pushers.get(user_id, {})

+42

-7

synapse/replication/http/_base.py less more

105	105
106	106	assert self.METHOD in ("PUT", "POST", "GET")
107	107
	108	self._replication_secret = None
	109	if hs.config.worker.worker_replication_secret:
	110	self._replication_secret = hs.config.worker.worker_replication_secret
	111
	112	def _check_auth(self, request) -> None:
	113	# Get the authorization header.
	114	auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
	115
	116	if len(auth_headers) > 1:
	117	raise RuntimeError("Too many Authorization headers.")
	118	parts = auth_headers[0].split(b" ")
	119	if parts[0] == b"Bearer" and len(parts) == 2:
	120	received_secret = parts[1].decode("ascii")
	121	if self._replication_secret == received_secret:
	122	# Success!
	123	return
	124
	125	raise RuntimeError("Invalid Authorization header.")
	126
108	127	@abc.abstractmethod
109	128	async def _serialize_payload(**kwargs):
110	129	"""Static method that is called when creating a request.

148	167	instance_map = hs.config.worker.instance_map
149	168
150	169	outgoing_gauge = _pending_outgoing_requests.labels(cls.NAME)
	170
	171	replication_secret = None
	172	if hs.config.worker.worker_replication_secret:
	173	replication_secret = hs.config.worker.worker_replication_secret.encode(
	174	"ascii"
	175	)
151	176
152	177	@trace(opname="outgoing_replication_request")
153	178	@outgoing_gauge.track_inprogress()

201	226	# the master, and so whether we should clean up or not.
202	227	while True:
203	228	headers = {} # type: Dict[bytes, List[bytes]]
	229	# Add an authorization header, if configured.
	230	if replication_secret:
	231	headers[b"Authorization"] = [b"Bearer " + replication_secret]
204	232	inject_active_span_byte_dict(headers, None, check_destination=False)
205	233	try:
206	234	result = await request_func(uri, data, headers=headers)

235	263	"""
236	264
237	265	url_args = list(self.PATH_ARGS)
238		handler = self._handle_request
239	266	method = self.METHOD
240	267
241	268	if self.CACHE:
242		handler = self._cached_handler # type: ignore
243	269	url_args.append("txn_id")
244	270
245	271	args = "/".join("(?P<%s>[^/]+)" % (arg,) for arg in url_args)
246	272	pattern = re.compile("^/_synapse/replication/%s/%s$" % (self.NAME, args))
247	273
248	274	http_server.register_paths(
249		method, [pattern], handler, self.__class__.__name__,
	275	method, [pattern], self._check_auth_and_handle, self.__class__.__name__,
250	276	)
251	277
252		def _cached_handler(self, request, txn_id, **kwargs):
	278	def _check_auth_and_handle(self, request, **kwargs):
253	279	"""Called on new incoming requests when caching is enabled. Checks
254	280	if there is a cached response for the request and returns that,
255	281	otherwise calls `_handle_request` and caches its response.

257	283	# We just use the txn_id here, but we probably also want to use the
258	284	# other PATH_ARGS as well.
259	285
260		assert self.CACHE
261
262		return self.response_cache.wrap(txn_id, self._handle_request, request, **kwargs)
	286	# Check the authorization headers before handling the request.
	287	if self._replication_secret:
	288	self._check_auth(request)
	289
	290	if self.CACHE:
	291	txn_id = kwargs.pop("txn_id")
	292
	293	return self.response_cache.wrap(
	294	txn_id, self._handle_request, request, **kwargs
	295	)
	296
	297	return self._handle_request(request, **kwargs)

+10

-2

synapse/replication/http/login.py less more

35	35	self.registration_handler = hs.get_registration_handler()
36	36
37	37	@staticmethod
38		async def _serialize_payload(user_id, device_id, initial_display_name, is_guest):
	38	async def _serialize_payload(
	39	user_id, device_id, initial_display_name, is_guest, is_appservice_ghost
	40	):
39	41	"""
40	42	Args:
41	43	device_id (str\|None): Device ID to use, if None a new one is

47	49	"device_id": device_id,
48	50	"initial_display_name": initial_display_name,
49	51	"is_guest": is_guest,
	52	"is_appservice_ghost": is_appservice_ghost,
50	53	}
51	54
52	55	async def _handle_request(self, request, user_id):

55	58	device_id = content["device_id"]
56	59	initial_display_name = content["initial_display_name"]
57	60	is_guest = content["is_guest"]
	61	is_appservice_ghost = content["is_appservice_ghost"]
58	62
59	63	device_id, access_token = await self.registration_handler.register_device(
60		user_id, device_id, initial_display_name, is_guest
	64	user_id,
	65	device_id,
	66	initial_display_name,
	67	is_guest,
	68	is_appservice_ghost=is_appservice_ghost,
61	69	)
62	70
63	71	return 200, {"device_id": device_id, "access_token": access_token}

+15

-5

synapse/replication/slave/storage/_slaved_id_tracker.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
	14	from typing import List, Optional, Tuple
14	15
	16	from synapse.storage.types import Connection
15	17	from synapse.storage.util.id_generators import _load_current_id
16	18
17	19
18	20	class SlavedIdTracker:
19		def __init__(self, db_conn, table, column, extra_tables=[], step=1):
	21	def __init__(
	22	self,
	23	db_conn: Connection,
	24	table: str,
	25	column: str,
	26	extra_tables: Optional[List[Tuple[str, str]]] = None,
	27	step: int = 1,
	28	):
20	29	self.step = step
21	30	self._current = _load_current_id(db_conn, table, column, step)
22		for table, column in extra_tables:
23		self.advance(None, _load_current_id(db_conn, table, column))
	31	if extra_tables:
	32	for table, column in extra_tables:
	33	self.advance(None, _load_current_id(db_conn, table, column))
24	34
25		def advance(self, instance_name, new_id):
	35	def advance(self, instance_name: Optional[str], new_id: int):
26	36	self._current = (max if self.step > 0 else min)(self._current, new_id)
27	37
28		def get_current_token(self):
	38	def get_current_token(self) -> int:
29	39	"""
30	40
31	41	Returns:

+12

-5

synapse/replication/slave/storage/pushers.py less more

12	12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
	15	from typing import TYPE_CHECKING
15	16
16	17	from synapse.replication.tcp.streams import PushersStream
17	18	from synapse.storage.database import DatabasePool
18	19	from synapse.storage.databases.main.pusher import PusherWorkerStore
	20	from synapse.storage.types import Connection
19	21
20	22	from ._base import BaseSlavedStore
21	23	from ._slaved_id_tracker import SlavedIdTracker
22	24
	25	if TYPE_CHECKING:
	26	from synapse.app.homeserver import HomeServer
	27
23	28
24	29	class SlavedPusherStore(PusherWorkerStore, BaseSlavedStore):
25		def __init__(self, database: DatabasePool, db_conn, hs):
	30	def __init__(self, database: DatabasePool, db_conn: Connection, hs: "HomeServer"):
26	31	super().__init__(database, db_conn, hs)
27		self._pushers_id_gen = SlavedIdTracker(
	32	self._pushers_id_gen = SlavedIdTracker( # type: ignore
28	33	db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
29	34	)
30	35
31		def get_pushers_stream_token(self):
	36	def get_pushers_stream_token(self) -> int:
32	37	return self._pushers_id_gen.get_current_token()
33	38
34		def process_replication_rows(self, stream_name, instance_name, token, rows):
	39	def process_replication_rows(
	40	self, stream_name: str, instance_name: str, token, rows
	41	) -> None:
35	42	if stream_name == PushersStream.NAME:
36		self._pushers_id_gen.advance(instance_name, token)
	43	self._pushers_id_gen.advance(instance_name, token) # type: ignore
37	44	return super().process_replication_rows(stream_name, instance_name, token, rows)

+1

-2

synapse/replication/tcp/protocol.py less more

171	171	# a logcontext which we use for processing incoming commands. We declare it as a
172	172	# background process so that the CPU stats get reported to prometheus.
173	173	ctx_name = "replication-conn-%s" % self.conn_id
174		self._logging_context = BackgroundProcessLoggingContext(ctx_name)
175		self._logging_context.request = ctx_name
	174	self._logging_context = BackgroundProcessLoggingContext(ctx_name, ctx_name)
176	175
177	176	def connectionMade(self):
178	177	logger.info("[%s] Connection established", self.id())

+1

-1

synapse/res/templates/notif.html less more

28	28	{{ message.body_text_html }}
29	29	{%- elif message.msgtype == "m.notice" %}
30	30	{{ message.body_text_html }}
31		{%- elif message.msgtype == "m.image" %}
	31	{%- elif message.msgtype == "m.image" and message.image_url %}
32	32	<img src="{{ message.image_url\|mxc_to_http(640, 480, scale) }}" />
33	33	{%- elif message.msgtype == "m.file" %}
34	34	<span class="filename">{{ message.body_text_plain }}</span>

+19

-0

synapse/res/username_picker/index.html less more

	0	<!DOCTYPE html>
	1	<html lang="en">
	2	<head>
	3	<title>Synapse Login</title>
	4	<link rel="stylesheet" href="style.css" type="text/css" />
	5	</head>
	6	<body>
	7	<div class="card">
	8	<form method="post" class="form__input" id="form" action="submit">
	9	<label for="field-username">Please pick your username:</label>
	10	<input type="text" name="username" id="field-username" autofocus="">
	11	<input type="submit" class="button button--full-width" id="button-submit" value="Submit">
	12	</form>
	13	<!-- this is used for feedback -->
	14	<div role=alert class="tooltip hidden" id="message"></div>
	15	<script src="script.js"></script>
	16	</div>
	17	</body>
	18	</html>

+95

-0

synapse/res/username_picker/script.js less more

	0	let inputField = document.getElementById("field-username");
	1	let inputForm = document.getElementById("form");
	2	let submitButton = document.getElementById("button-submit");
	3	let message = document.getElementById("message");
	4
	5	// Submit username and receive response
	6	function showMessage(messageText) {
	7	// Unhide the message text
	8	message.classList.remove("hidden");
	9
	10	message.textContent = messageText;
	11	};
	12
	13	function doSubmit() {
	14	showMessage("Success. Please wait a moment for your browser to redirect.");
	15
	16	// remove the event handler before re-submitting the form.
	17	delete inputForm.onsubmit;
	18	inputForm.submit();
	19	}
	20
	21	function onResponse(response) {
	22	// Display message
	23	showMessage(response);
	24
	25	// Enable submit button and input field
	26	submitButton.classList.remove('button--disabled');
	27	submitButton.value = "Submit";
	28	};
	29
	30	let allowedUsernameCharacters = RegExp("[^a-z0-9\\.\\_\\=\\-\\/]");
	31	function usernameIsValid(username) {
	32	return !allowedUsernameCharacters.test(username);
	33	}
	34	let allowedCharactersString = "lowercase letters, digits, ., _, -, /, =";
	35
	36	function buildQueryString(params) {
	37	return Object.keys(params)
	38	.map(k => encodeURIComponent(k) + '=' + encodeURIComponent(params[k]))
	39	.join('&');
	40	}
	41
	42	function submitUsername(username) {
	43	if(username.length == 0) {
	44	onResponse("Please enter a username.");
	45	return;
	46	}
	47	if(!usernameIsValid(username)) {
	48	onResponse("Invalid username. Only the following characters are allowed: " + allowedCharactersString);
	49	return;
	50	}
	51
	52	// if this browser doesn't support fetch, skip the availability check.
	53	if(!window.fetch) {
	54	doSubmit();
	55	return;
	56	}
	57
	58	let check_uri = 'check?' + buildQueryString({"username": username});
	59	fetch(check_uri, {
	60	// include the cookie
	61	"credentials": "same-origin",
	62	}).then((response) => {
	63	if(!response.ok) {
	64	// for non-200 responses, raise the body of the response as an exception
	65	return response.text().then((text) => { throw text; });
	66	} else {
	67	return response.json();
	68	}
	69	}).then((json) => {
	70	if(json.error) {
	71	throw json.error;
	72	} else if(json.available) {
	73	doSubmit();
	74	} else {
	75	onResponse("This username is not available, please choose another.");
	76	}
	77	}).catch((err) => {
	78	onResponse("Error checking username availability: " + err);
	79	});
	80	}
	81
	82	function clickSubmit() {
	83	event.preventDefault();
	84	if(submitButton.classList.contains('button--disabled')) { return; }
	85
	86	// Disable submit button and input field
	87	submitButton.classList.add('button--disabled');
	88
	89	// Submit username
	90	submitButton.value = "Checking...";
	91	submitUsername(inputField.value);
	92	};
	93
	94	inputForm.onsubmit = clickSubmit;

+27

-0

synapse/res/username_picker/style.css less more

	0	input[type="text"] {
	1	font-size: 100%;
	2	background-color: #ededf0;
	3	border: 1px solid #fff;
	4	border-radius: .2em;
	5	padding: .5em .9em;
	6	display: block;
	7	width: 26em;
	8	}
	9
	10	.button--disabled {
	11	border-color: #fff;
	12	background-color: transparent;
	13	color: #000;
	14	text-transform: none;
	15	}
	16
	17	.hidden {
	18	display: none;
	19	}
	20
	21	.tooltip {
	22	background-color: #f9f9fa;
	23	padding: 1em;
	24	margin: 1em 0;
	25	}
	26

+2

-0

synapse/rest/admin/__init__.py less more

37	37	DeleteRoomRestServlet,
38	38	JoinRoomAliasServlet,
39	39	ListRoomRestServlet,
	40	MakeRoomAdminRestServlet,
40	41	RoomMembersRestServlet,
41	42	RoomRestServlet,
42	43	ShutdownRoomRestServlet,

227	228	EventReportDetailRestServlet(hs).register(http_server)
228	229	EventReportsRestServlet(hs).register(http_server)
229	230	PushersRestServlet(hs).register(http_server)
	231	MakeRoomAdminRestServlet(hs).register(http_server)
230	232
231	233
232	234	def register_servlets_for_client_rest_resource(hs, http_server):

+167

-19

synapse/rest/admin/rooms.py less more

13	13	# limitations under the License.
14	14	import logging
15	15	from http import HTTPStatus
16		from typing import List, Optional
17
18		from synapse.api.constants import EventTypes, JoinRules
19		from synapse.api.errors import Codes, NotFoundError, SynapseError
	16	from typing import TYPE_CHECKING, List, Optional, Tuple
	17
	18	from synapse.api.constants import EventTypes, JoinRules, Membership
	19	from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
20	20	from synapse.http.servlet import (
21	21	RestServlet,
22	22	assert_params_in_dict,

24	24	parse_json_object_from_request,
25	25	parse_string,
26	26	)
	27	from synapse.http.site import SynapseRequest
27	28	from synapse.rest.admin._base import (
28	29	admin_patterns,
29	30	assert_requester_is_admin,
30	31	assert_user_is_admin,
31	32	)
32	33	from synapse.storage.databases.main.room import RoomSortOrder
33		from synapse.types import RoomAlias, RoomID, UserID, create_requester
	34	from synapse.types import JsonDict, RoomAlias, RoomID, UserID, create_requester
	35
	36	if TYPE_CHECKING:
	37	from synapse.server import HomeServer
	38
34	39
35	40	logger = logging.getLogger(__name__)
36	41

44	49
45	50	PATTERNS = admin_patterns("/shutdown_room/(?P<room_id>[^/]+)")
46	51
47		def __init__(self, hs):
	52	def __init__(self, hs: "HomeServer"):
48	53	self.hs = hs
49	54	self.auth = hs.get_auth()
50	55	self.room_shutdown_handler = hs.get_room_shutdown_handler()
51	56
52		async def on_POST(self, request, room_id):
	57	async def on_POST(
	58	self, request: SynapseRequest, room_id: str
	59	) -> Tuple[int, JsonDict]:
53	60	requester = await self.auth.get_user_by_req(request)
54	61	await assert_user_is_admin(self.auth, requester.user)
55	62

85	92
86	93	PATTERNS = admin_patterns("/rooms/(?P<room_id>[^/]+)/delete$")
87	94
88		def __init__(self, hs):
	95	def __init__(self, hs: "HomeServer"):
89	96	self.hs = hs
90	97	self.auth = hs.get_auth()
91	98	self.room_shutdown_handler = hs.get_room_shutdown_handler()
92	99	self.pagination_handler = hs.get_pagination_handler()
93	100
94		async def on_POST(self, request, room_id):
	101	async def on_POST(
	102	self, request: SynapseRequest, room_id: str
	103	) -> Tuple[int, JsonDict]:
95	104	requester = await self.auth.get_user_by_req(request)
96	105	await assert_user_is_admin(self.auth, requester.user)
97	106

145	154
146	155	PATTERNS = admin_patterns("/rooms$")
147	156
148		def __init__(self, hs):
	157	def __init__(self, hs: "HomeServer"):
149	158	self.store = hs.get_datastore()
150	159	self.auth = hs.get_auth()
151	160	self.admin_handler = hs.get_admin_handler()
152	161
153		async def on_GET(self, request):
	162	async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
154	163	requester = await self.auth.get_user_by_req(request)
155	164	await assert_user_is_admin(self.auth, requester.user)
156	165

235	244
236	245	PATTERNS = admin_patterns("/rooms/(?P<room_id>[^/]+)$")
237	246
238		def __init__(self, hs):
	247	def __init__(self, hs: "HomeServer"):
239	248	self.hs = hs
240	249	self.auth = hs.get_auth()
241	250	self.store = hs.get_datastore()
242	251
243		async def on_GET(self, request, room_id):
	252	async def on_GET(
	253	self, request: SynapseRequest, room_id: str
	254	) -> Tuple[int, JsonDict]:
244	255	await assert_requester_is_admin(self.auth, request)
245	256
246	257	ret = await self.store.get_room_with_stats(room_id)
247	258	if not ret:
248	259	raise NotFoundError("Room not found")
249	260
250		return 200, ret
	261	members = await self.store.get_users_in_room(room_id)
	262	ret["joined_local_devices"] = await self.store.count_devices_by_users(members)
	263
	264	return (200, ret)
251	265
252	266
253	267	class RoomMembersRestServlet(RestServlet):

257	271
258	272	PATTERNS = admin_patterns("/rooms/(?P<room_id>[^/]+)/members")
259	273
260		def __init__(self, hs):
	274	def __init__(self, hs: "HomeServer"):
261	275	self.hs = hs
262	276	self.auth = hs.get_auth()
263	277	self.store = hs.get_datastore()
264	278
265		async def on_GET(self, request, room_id):
	279	async def on_GET(
	280	self, request: SynapseRequest, room_id: str
	281	) -> Tuple[int, JsonDict]:
266	282	await assert_requester_is_admin(self.auth, request)
267	283
268	284	ret = await self.store.get_room(room_id)

279	295
280	296	PATTERNS = admin_patterns("/join/(?P<room_identifier>[^/]*)")
281	297
282		def __init__(self, hs):
	298	def __init__(self, hs: "HomeServer"):
283	299	self.hs = hs
284	300	self.auth = hs.get_auth()
285	301	self.room_member_handler = hs.get_room_member_handler()
286	302	self.admin_handler = hs.get_admin_handler()
287	303	self.state_handler = hs.get_state_handler()
288	304
289		async def on_POST(self, request, room_identifier):
	305	async def on_POST(
	306	self, request: SynapseRequest, room_identifier: str
	307	) -> Tuple[int, JsonDict]:
290	308	requester = await self.auth.get_user_by_req(request)
291	309	await assert_user_is_admin(self.auth, requester.user)
292	310

313	331	handler = self.room_member_handler
314	332	room_alias = RoomAlias.from_string(room_identifier)
315	333	room_id, remote_room_hosts = await handler.lookup_room_alias(room_alias)
316		room_id = room_id.to_string()
317	334	else:
318	335	raise SynapseError(
319	336	400, "%s was not legal room ID or room alias" % (room_identifier,)

350	367	)
351	368
352	369	return 200, {"room_id": room_id}
	370
	371
	372	class MakeRoomAdminRestServlet(RestServlet):
	373	"""Allows a server admin to get power in a room if a local user has power in
	374	a room. Will also invite the user if they're not in the room and it's a
	375	private room. Can specify another user (rather than the admin user) to be
	376	granted power, e.g.:
	377
	378	POST/_synapse/admin/v1/rooms/<room_id_or_alias>/make_room_admin
	379	{
	380	"user_id": "@foo:example.com"
	381	}
	382	"""
	383
	384	PATTERNS = admin_patterns("/rooms/(?P<room_identifier>[^/]*)/make_room_admin")
	385
	386	def __init__(self, hs: "HomeServer"):
	387	self.hs = hs
	388	self.auth = hs.get_auth()
	389	self.room_member_handler = hs.get_room_member_handler()
	390	self.event_creation_handler = hs.get_event_creation_handler()
	391	self.state_handler = hs.get_state_handler()
	392	self.is_mine_id = hs.is_mine_id
	393
	394	async def on_POST(self, request, room_identifier):
	395	requester = await self.auth.get_user_by_req(request)
	396	await assert_user_is_admin(self.auth, requester.user)
	397	content = parse_json_object_from_request(request, allow_empty_body=True)
	398
	399	# Resolve to a room ID, if necessary.
	400	if RoomID.is_valid(room_identifier):
	401	room_id = room_identifier
	402	elif RoomAlias.is_valid(room_identifier):
	403	room_alias = RoomAlias.from_string(room_identifier)
	404	room_id, _ = await self.room_member_handler.lookup_room_alias(room_alias)
	405	room_id = room_id.to_string()
	406	else:
	407	raise SynapseError(
	408	400, "%s was not legal room ID or room alias" % (room_identifier,)
	409	)
	410
	411	# Which user to grant room admin rights to.
	412	user_to_add = content.get("user_id", requester.user.to_string())
	413
	414	# Figure out which local users currently have power in the room, if any.
	415	room_state = await self.state_handler.get_current_state(room_id)
	416	if not room_state:
	417	raise SynapseError(400, "Server not in room")
	418
	419	create_event = room_state[(EventTypes.Create, "")]
	420	power_levels = room_state.get((EventTypes.PowerLevels, ""))
	421
	422	if power_levels is not None:
	423	# We pick the local user with the highest power.
	424	user_power = power_levels.content.get("users", {})
	425	admin_users = [
	426	user_id for user_id in user_power if self.is_mine_id(user_id)
	427	]
	428	admin_users.sort(key=lambda user: user_power[user])
	429
	430	if not admin_users:
	431	raise SynapseError(400, "No local admin user in room")
	432
	433	admin_user_id = admin_users[-1]
	434
	435	pl_content = power_levels.content
	436	else:
	437	# If there is no power level events then the creator has rights.
	438	pl_content = {}
	439	admin_user_id = create_event.sender
	440	if not self.is_mine_id(admin_user_id):
	441	raise SynapseError(
	442	400, "No local admin user in room",
	443	)
	444
	445	# Grant the user power equal to the room admin by attempting to send an
	446	# updated power level event.
	447	new_pl_content = dict(pl_content)
	448	new_pl_content["users"] = dict(pl_content.get("users", {}))
	449	new_pl_content["users"][user_to_add] = new_pl_content["users"][admin_user_id]
	450
	451	fake_requester = create_requester(
	452	admin_user_id, authenticated_entity=requester.authenticated_entity,
	453	)
	454
	455	try:
	456	await self.event_creation_handler.create_and_send_nonmember_event(
	457	fake_requester,
	458	event_dict={
	459	"content": new_pl_content,
	460	"sender": admin_user_id,
	461	"type": EventTypes.PowerLevels,
	462	"state_key": "",
	463	"room_id": room_id,
	464	},
	465	)
	466	except AuthError:
	467	# The admin user we found turned out not to have enough power.
	468	raise SynapseError(
	469	400, "No local admin user in room with power to update power levels."
	470	)
	471
	472	# Now we check if the user we're granting admin rights to is already in
	473	# the room. If not and it's not a public room we invite them.
	474	member_event = room_state.get((EventTypes.Member, user_to_add))
	475	is_joined = False
	476	if member_event:
	477	is_joined = member_event.content["membership"] in (
	478	Membership.JOIN,
	479	Membership.INVITE,
	480	)
	481
	482	if is_joined:
	483	return 200, {}
	484
	485	join_rules = room_state.get((EventTypes.JoinRules, ""))
	486	is_public = False
	487	if join_rules:
	488	is_public = join_rules.content.get("join_rule") == JoinRules.PUBLIC
	489
	490	if is_public:
	491	return 200, {}
	492
	493	await self.room_member_handler.update_membership(
	494	fake_requester,
	495	target=UserID.from_string(user_to_add),
	496	room_id=room_id,
	497	action=Membership.INVITE,
	498	)
	499
	500	return 200, {}

+6

-17

synapse/rest/admin/users.py less more

40	40	from synapse.server import HomeServer
41	41
42	42	logger = logging.getLogger(__name__)
43
44		_GET_PUSHERS_ALLOWED_KEYS = {
45		"app_display_name",
46		"app_id",
47		"data",
48		"device_display_name",
49		"kind",
50		"lang",
51		"profile_tag",
52		"pushkey",
53		}
54	43
55	44
56	45	class UsersRestServlet(RestServlet):

319	308	data={},
320	309	)
321	310
322		if "avatar_url" in body and type(body["avatar_url"]) == str:
	311	if "avatar_url" in body and isinstance(body["avatar_url"], str):
323	312	await self.profile_handler.set_avatar_url(
324		user_id, requester, body["avatar_url"], True
	313	target_user, requester, body["avatar_url"], True
325	314	)
326	315
327	316	ret = await self.admin_handler.get_user(target_user)

418	407
419	408	if user_type is not None and user_type not in UserTypes.ALL_USER_TYPES:
420	409	raise SynapseError(400, "Invalid user type")
	410
	411	if "mac" not in body:
	412	raise SynapseError(400, "mac must be specified", errcode=Codes.BAD_JSON)
421	413
422	414	got_mac = body["mac"]
423	415

766	758
767	759	pushers = await self.store.get_pushers_by_user_id(user_id)
768	760
769		filtered_pushers = [
770		{k: v for k, v in p.items() if k in _GET_PUSHERS_ALLOWED_KEYS}
771		for p in pushers
772		]
	761	filtered_pushers = [p.as_dict() for p in pushers]
773	762
774	763	return 200, {"pushers": filtered_pushers, "total": len(filtered_pushers)}
775	764

+19

-6

synapse/rest/client/v1/login.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16		from typing import Awaitable, Callable, Dict, Optional
	16	from typing import TYPE_CHECKING, Awaitable, Callable, Dict, Optional
17	17
18	18	from synapse.api.errors import Codes, LoginError, SynapseError
19	19	from synapse.api.ratelimiting import Ratelimiter

29	29	from synapse.rest.well_known import WellKnownBuilder
30	30	from synapse.types import JsonDict, UserID
31	31
	32	if TYPE_CHECKING:
	33	from synapse.server import HomeServer
	34
32	35	logger = logging.getLogger(__name__)
33	36
34	37

41	44	JWT_TYPE_DEPRECATED = "m.login.jwt"
42	45	APPSERVICE_TYPE = "uk.half-shot.msc2778.login.application_service"
43	46
44		def __init__(self, hs):
	47	def __init__(self, hs: "HomeServer"):
45	48	super().__init__()
46	49	self.hs = hs
47	50

104	107	return 200, {"flows": flows}
105	108
106	109	async def on_POST(self, request: SynapseRequest):
107		self._address_ratelimiter.ratelimit(request.getClientIP())
108
109	110	login_submission = parse_json_object_from_request(request)
110	111
111	112	try:
112	113	if login_submission["type"] == LoginRestServlet.APPSERVICE_TYPE:
113	114	appservice = self.auth.get_appservice_by_req(request)
	115
	116	if appservice.is_rate_limited():
	117	self._address_ratelimiter.ratelimit(request.getClientIP())
	118
114	119	result = await self._do_appservice_login(login_submission, appservice)
115	120	elif self.jwt_enabled and (
116	121	login_submission["type"] == LoginRestServlet.JWT_TYPE
117	122	or login_submission["type"] == LoginRestServlet.JWT_TYPE_DEPRECATED
118	123	):
	124	self._address_ratelimiter.ratelimit(request.getClientIP())
119	125	result = await self._do_jwt_login(login_submission)
120	126	elif login_submission["type"] == LoginRestServlet.TOKEN_TYPE:
	127	self._address_ratelimiter.ratelimit(request.getClientIP())
121	128	result = await self._do_token_login(login_submission)
122	129	else:
	130	self._address_ratelimiter.ratelimit(request.getClientIP())
123	131	result = await self._do_other_login(login_submission)
124	132	except KeyError:
125	133	raise SynapseError(400, "Missing JSON keys.")

158	166	if not appservice.is_interested_in_user(qualified_user_id):
159	167	raise LoginError(403, "Invalid access_token", errcode=Codes.FORBIDDEN)
160	168
161		return await self._complete_login(qualified_user_id, login_submission)
	169	return await self._complete_login(
	170	qualified_user_id, login_submission, ratelimit=appservice.is_rate_limited()
	171	)
162	172
163	173	async def _do_other_login(self, login_submission: JsonDict) -> Dict[str, str]:
164	174	"""Handle non-token/saml/jwt logins

193	203	login_submission: JsonDict,
194	204	callback: Optional[Callable[[Dict[str, str]], Awaitable[None]]] = None,
195	205	create_non_existent_users: bool = False,
	206	ratelimit: bool = True,
196	207	) -> Dict[str, str]:
197	208	"""Called when we've successfully authed the user and now need to
198	209	actually login them in (e.g. create devices). This gets called on

207	218	callback: Callback function to run after login.
208	219	create_non_existent_users: Whether to create the user if they don't
209	220	exist. Defaults to False.
	221	ratelimit: Whether to ratelimit the login request.
210	222
211	223	Returns:
212	224	result: Dictionary of account information after successful login.

215	227	# Before we actually log them in we check if they've already logged in
216	228	# too often. This happens here rather than before as we don't
217	229	# necessarily know the user before now.
218		self._account_ratelimiter.ratelimit(user_id.lower())
	230	if ratelimit:
	231	self._account_ratelimiter.ratelimit(user_id.lower())
219	232
220	233	if create_non_existent_users:
221	234	canonical_uid = await self.auth_handler.check_user_exists(user_id)

+1

-14

synapse/rest/client/v1/pusher.py less more

27	27
28	28	logger = logging.getLogger(__name__)
29	29
30		ALLOWED_KEYS = {
31		"app_display_name",
32		"app_id",
33		"data",
34		"device_display_name",
35		"kind",
36		"lang",
37		"profile_tag",
38		"pushkey",
39		}
40
41	30
42	31	class PushersRestServlet(RestServlet):
43	32	PATTERNS = client_patterns("/pushers$", v1=True)

53	42
54	43	pushers = await self.hs.get_datastore().get_pushers_by_user_id(user.to_string())
55	44
56		filtered_pushers = [
57		{k: v for k, v in p.items() if k in ALLOWED_KEYS} for p in pushers
58		]
	45	filtered_pushers = [p.as_dict() for p in pushers]
59	46
60	47	return 200, {"pushers": filtered_pushers}
61	48

+10

-7

synapse/rest/client/v1/room.py less more

962	962	)
963	963
964	964
965		def register_servlets(hs, http_server):
	965	def register_servlets(hs, http_server, is_worker=False):
966	966	RoomStateEventRestServlet(hs).register(http_server)
967		RoomCreateRestServlet(hs).register(http_server)
968	967	RoomMemberListRestServlet(hs).register(http_server)
969	968	JoinedRoomMemberListRestServlet(hs).register(http_server)
970	969	RoomMessageListRestServlet(hs).register(http_server)
971	970	JoinRoomAliasServlet(hs).register(http_server)
972		RoomForgetRestServlet(hs).register(http_server)
973	971	RoomMembershipRestServlet(hs).register(http_server)
974	972	RoomSendEventRestServlet(hs).register(http_server)
975	973	PublicRoomListRestServlet(hs).register(http_server)
976	974	RoomStateRestServlet(hs).register(http_server)
977	975	RoomRedactEventRestServlet(hs).register(http_server)
978	976	RoomTypingRestServlet(hs).register(http_server)
979		SearchRestServlet(hs).register(http_server)
980		JoinedRoomsRestServlet(hs).register(http_server)
981		RoomEventServlet(hs).register(http_server)
982	977	RoomEventContextServlet(hs).register(http_server)
983		RoomAliasListServlet(hs).register(http_server)
	978
	979	# Some servlets only get registered for the main process.
	980	if not is_worker:
	981	RoomCreateRestServlet(hs).register(http_server)
	982	RoomForgetRestServlet(hs).register(http_server)
	983	SearchRestServlet(hs).register(http_server)
	984	JoinedRoomsRestServlet(hs).register(http_server)
	985	RoomEventServlet(hs).register(http_server)
	986	RoomAliasListServlet(hs).register(http_server)
984	987
985	988
986	989	def register_deprecated_servlets(hs, http_server):

+7

-3

synapse/rest/client/v2_alpha/account.py less more

253	253	logger.error("Auth succeeded but no known type! %r", result.keys())
254	254	raise SynapseError(500, "", Codes.UNKNOWN)
255	255
256		# If we have a password in this request, prefer it. Otherwise, there
257		# must be a password hash from an earlier request.
	256	# If we have a password in this request, prefer it. Otherwise, use the
	257	# password hash from an earlier request.
258	258	if new_password:
259	259	password_hash = await self.auth_handler.hash(new_password)
260		else:
	260	elif session_id is not None:
261	261	password_hash = await self.auth_handler.get_session_data(
262	262	session_id, "password_hash", None
263	263	)
	264	else:
	265	# UI validation was skipped, but the request did not include a new
	266	# password.
	267	password_hash = None
264	268	if not password_hash:
265	269	raise SynapseError(400, "Missing params: password", Codes.MISSING_PARAM)
266	270

+45

-3

synapse/rest/client/v2_alpha/groups.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
	17	from functools import wraps
17	18
18	19	from synapse.api.errors import SynapseError
19	20	from synapse.http.servlet import RestServlet, parse_json_object_from_request

24	25	logger = logging.getLogger(__name__)
25	26
26	27
	28	def _validate_group_id(f):
	29	"""Wrapper to validate the form of the group ID.
	30
	31	Can be applied to any on_FOO methods that accepts a group ID as a URL parameter.
	32	"""
	33
	34	@wraps(f)
	35	def wrapper(self, request, group_id, args, *kwargs):
	36	if not GroupID.is_valid(group_id):
	37	raise SynapseError(400, "%s is not a legal group ID" % (group_id,))
	38
	39	return f(self, request, group_id, args, *kwargs)
	40
	41	return wrapper
	42
	43
27	44	class GroupServlet(RestServlet):
28	45	"""Get the group profile
29	46	"""

36	53	self.clock = hs.get_clock()
37	54	self.groups_handler = hs.get_groups_local_handler()
38	55
	56	@_validate_group_id
39	57	async def on_GET(self, request, group_id):
40	58	requester = await self.auth.get_user_by_req(request, allow_guest=True)
41	59	requester_user_id = requester.user.to_string()

46	64
47	65	return 200, group_description
48	66
	67	@_validate_group_id
49	68	async def on_POST(self, request, group_id):
50	69	requester = await self.auth.get_user_by_req(request)
51	70	requester_user_id = requester.user.to_string()

70	89	self.clock = hs.get_clock()
71	90	self.groups_handler = hs.get_groups_local_handler()
72	91
	92	@_validate_group_id
73	93	async def on_GET(self, request, group_id):
74	94	requester = await self.auth.get_user_by_req(request, allow_guest=True)
75	95	requester_user_id = requester.user.to_string()

101	121	self.clock = hs.get_clock()
102	122	self.groups_handler = hs.get_groups_local_handler()
103	123
	124	@_validate_group_id
104	125	async def on_PUT(self, request, group_id, category_id, room_id):
105	126	requester = await self.auth.get_user_by_req(request)
106	127	requester_user_id = requester.user.to_string()

116	137
117	138	return 200, resp
118	139
	140	@_validate_group_id
119	141	async def on_DELETE(self, request, group_id, category_id, room_id):
120	142	requester = await self.auth.get_user_by_req(request)
121	143	requester_user_id = requester.user.to_string()

141	163	self.clock = hs.get_clock()
142	164	self.groups_handler = hs.get_groups_local_handler()
143	165
	166	@_validate_group_id
144	167	async def on_GET(self, request, group_id, category_id):
145	168	requester = await self.auth.get_user_by_req(request, allow_guest=True)
146	169	requester_user_id = requester.user.to_string()

151	174
152	175	return 200, category
153	176
	177	@_validate_group_id
154	178	async def on_PUT(self, request, group_id, category_id):
155	179	requester = await self.auth.get_user_by_req(request)
156	180	requester_user_id = requester.user.to_string()

162	186
163	187	return 200, resp
164	188
	189	@_validate_group_id
165	190	async def on_DELETE(self, request, group_id, category_id):
166	191	requester = await self.auth.get_user_by_req(request)
167	192	requester_user_id = requester.user.to_string()

185	210	self.clock = hs.get_clock()
186	211	self.groups_handler = hs.get_groups_local_handler()
187	212
	213	@_validate_group_id
188	214	async def on_GET(self, request, group_id):
189	215	requester = await self.auth.get_user_by_req(request, allow_guest=True)
190	216	requester_user_id = requester.user.to_string()

208	234	self.clock = hs.get_clock()
209	235	self.groups_handler = hs.get_groups_local_handler()
210	236
	237	@_validate_group_id
211	238	async def on_GET(self, request, group_id, role_id):
212	239	requester = await self.auth.get_user_by_req(request, allow_guest=True)
213	240	requester_user_id = requester.user.to_string()

218	245
219	246	return 200, category
220	247
	248	@_validate_group_id
221	249	async def on_PUT(self, request, group_id, role_id):
222	250	requester = await self.auth.get_user_by_req(request)
223	251	requester_user_id = requester.user.to_string()

229	257
230	258	return 200, resp
231	259
	260	@_validate_group_id
232	261	async def on_DELETE(self, request, group_id, role_id):
233	262	requester = await self.auth.get_user_by_req(request)
234	263	requester_user_id = requester.user.to_string()

252	281	self.clock = hs.get_clock()
253	282	self.groups_handler = hs.get_groups_local_handler()
254	283
	284	@_validate_group_id
255	285	async def on_GET(self, request, group_id):
256	286	requester = await self.auth.get_user_by_req(request, allow_guest=True)
257	287	requester_user_id = requester.user.to_string()

283	313	self.clock = hs.get_clock()
284	314	self.groups_handler = hs.get_groups_local_handler()
285	315
	316	@_validate_group_id
286	317	async def on_PUT(self, request, group_id, role_id, user_id):
287	318	requester = await self.auth.get_user_by_req(request)
288	319	requester_user_id = requester.user.to_string()

298	329
299	330	return 200, resp
300	331
	332	@_validate_group_id
301	333	async def on_DELETE(self, request, group_id, role_id, user_id):
302	334	requester = await self.auth.get_user_by_req(request)
303	335	requester_user_id = requester.user.to_string()

321	353	self.clock = hs.get_clock()
322	354	self.groups_handler = hs.get_groups_local_handler()
323	355
	356	@_validate_group_id
324	357	async def on_GET(self, request, group_id):
325	358	requester = await self.auth.get_user_by_req(request, allow_guest=True)
326	359	requester_user_id = requester.user.to_string()
327
328		if not GroupID.is_valid(group_id):
329		raise SynapseError(400, "%s was not legal group ID" % (group_id,))
330	360
331	361	result = await self.groups_handler.get_rooms_in_group(
332	362	group_id, requester_user_id

347	377	self.clock = hs.get_clock()
348	378	self.groups_handler = hs.get_groups_local_handler()
349	379
	380	@_validate_group_id
350	381	async def on_GET(self, request, group_id):
351	382	requester = await self.auth.get_user_by_req(request, allow_guest=True)
352	383	requester_user_id = requester.user.to_string()

370	401	self.clock = hs.get_clock()
371	402	self.groups_handler = hs.get_groups_local_handler()
372	403
	404	@_validate_group_id
373	405	async def on_GET(self, request, group_id):
374	406	requester = await self.auth.get_user_by_req(request)
375	407	requester_user_id = requester.user.to_string()

392	424	self.auth = hs.get_auth()
393	425	self.groups_handler = hs.get_groups_local_handler()
394	426
	427	@_validate_group_id
395	428	async def on_PUT(self, request, group_id):
396	429	requester = await self.auth.get_user_by_req(request)
397	430	requester_user_id = requester.user.to_string()

448	481	self.clock = hs.get_clock()
449	482	self.groups_handler = hs.get_groups_local_handler()
450	483
	484	@_validate_group_id
451	485	async def on_PUT(self, request, group_id, room_id):
452	486	requester = await self.auth.get_user_by_req(request)
453	487	requester_user_id = requester.user.to_string()

459	493
460	494	return 200, result
461	495
	496	@_validate_group_id
462	497	async def on_DELETE(self, request, group_id, room_id):
463	498	requester = await self.auth.get_user_by_req(request)
464	499	requester_user_id = requester.user.to_string()

485	520	self.clock = hs.get_clock()
486	521	self.groups_handler = hs.get_groups_local_handler()
487	522
	523	@_validate_group_id
488	524	async def on_PUT(self, request, group_id, room_id, config_key):
489	525	requester = await self.auth.get_user_by_req(request)
490	526	requester_user_id = requester.user.to_string()

513	549	self.store = hs.get_datastore()
514	550	self.is_mine_id = hs.is_mine_id
515	551
	552	@_validate_group_id
516	553	async def on_PUT(self, request, group_id, user_id):
517	554	requester = await self.auth.get_user_by_req(request)
518	555	requester_user_id = requester.user.to_string()

540	577	self.clock = hs.get_clock()
541	578	self.groups_handler = hs.get_groups_local_handler()
542	579
	580	@_validate_group_id
543	581	async def on_PUT(self, request, group_id, user_id):
544	582	requester = await self.auth.get_user_by_req(request)
545	583	requester_user_id = requester.user.to_string()

564	602	self.clock = hs.get_clock()
565	603	self.groups_handler = hs.get_groups_local_handler()
566	604
	605	@_validate_group_id
567	606	async def on_PUT(self, request, group_id):
568	607	requester = await self.auth.get_user_by_req(request)
569	608	requester_user_id = requester.user.to_string()

588	627	self.clock = hs.get_clock()
589	628	self.groups_handler = hs.get_groups_local_handler()
590	629
	630	@_validate_group_id
591	631	async def on_PUT(self, request, group_id):
592	632	requester = await self.auth.get_user_by_req(request)
593	633	requester_user_id = requester.user.to_string()

612	652	self.clock = hs.get_clock()
613	653	self.groups_handler = hs.get_groups_local_handler()
614	654
	655	@_validate_group_id
615	656	async def on_PUT(self, request, group_id):
616	657	requester = await self.auth.get_user_by_req(request)
617	658	requester_user_id = requester.user.to_string()

636	677	self.clock = hs.get_clock()
637	678	self.store = hs.get_datastore()
638	679
	680	@_validate_group_id
639	681	async def on_PUT(self, request, group_id):
640	682	requester = await self.auth.get_user_by_req(request)
641	683	requester_user_id = requester.user.to_string()

+13

-5

synapse/rest/client/v2_alpha/register.py less more

450	450
451	451	# == Normal User Registration == (everyone else)
452	452	if not self._registration_enabled:
453		raise SynapseError(403, "Registration has been disabled")
	453	raise SynapseError(403, "Registration has been disabled", Codes.FORBIDDEN)
454	454
455	455	# For regular registration, convert the provided username to lowercase
456	456	# before attempting to register it. This should mean that people who try

654	654	user_id = await self.registration_handler.appservice_register(
655	655	username, as_token
656	656	)
657		return await self._create_registration_details(user_id, body)
658
659		async def _create_registration_details(self, user_id, params):
	657	return await self._create_registration_details(
	658	user_id, body, is_appservice_ghost=True,
	659	)
	660
	661	async def _create_registration_details(
	662	self, user_id, params, is_appservice_ghost=False
	663	):
660	664	"""Complete registration of newly-registered user
661	665
662	666	Allocates device_id if one was not given; also creates access_token.

673	677	device_id = params.get("device_id")
674	678	initial_display_name = params.get("initial_device_display_name")
675	679	device_id, access_token = await self.registration_handler.register_device(
676		user_id, device_id, initial_display_name, is_guest=False
	680	user_id,
	681	device_id,
	682	initial_display_name,
	683	is_guest=False,
	684	is_appservice_ghost=is_appservice_ghost,
677	685	)
678	686
679	687	result.update({"access_token": access_token, "device_id": device_id})

+2

-1

synapse/rest/client/v2_alpha/sendtodevice.py less more

16	16	from typing import Tuple
17	17
18	18	from synapse.http import servlet
19		from synapse.http.servlet import parse_json_object_from_request
	19	from synapse.http.servlet import assert_params_in_dict, parse_json_object_from_request
20	20	from synapse.logging.opentracing import set_tag, trace
21	21	from synapse.rest.client.transactions import HttpTransactionCache
22	22

53	53	requester = await self.auth.get_user_by_req(request, allow_guest=True)
54	54
55	55	content = parse_json_object_from_request(request)
	56	assert_params_in_dict(content, ("messages",))
56	57
57	58	sender_user_id = requester.user.to_string()
58	59

+5

-4

synapse/rest/key/v2/remote_key_resource.py less more

12	12	# limitations under the License.
13	13
14	14	import logging
15		from typing import Dict, Set
	15	from typing import Dict
16	16
17	17	from signedjson.sign import sign_json
18	18

141	141
142	142	time_now_ms = self.clock.time_msec()
143	143
144		cache_misses = {} # type: Dict[str, Set[str]]
	144	# Note that the value is unused.
	145	cache_misses = {} # type: Dict[str, Dict[str, int]]
145	146	for (server_name, key_id, from_server), results in cached.items():
146	147	results = [(result["ts_added_ms"], result) for result in results]
147	148
148	149	if not results and key_id is not None:
149		cache_misses.setdefault(server_name, set()).add(key_id)
	150	cache_misses.setdefault(server_name, {})[key_id] = 0
150	151	continue
151	152
152	153	if key_id is not None:

200	201	)
201	202
202	203	if miss:
203		cache_misses.setdefault(server_name, set()).add(key_id)
	204	cache_misses.setdefault(server_name, {})[key_id] = 0
204	205	# Cast to bytes since postgresql returns a memoryview.
205	206	json_results.add(bytes(most_recent_result["key_json"]))
206	207	else:

+5

-0

synapse/rest/media/v1/_base.py less more

153	153	# clients are smart enough to be happy with Cache-Control
154	154	request.setHeader(b"Cache-Control", b"public,max-age=86400,s-maxage=86400")
155	155	request.setHeader(b"Content-Length", b"%d" % (file_size,))
	156
	157	# Tell web crawlers to not index, archive, or follow links in media. This
	158	# should help to prevent things in the media repo from showing up in web
	159	# search results.
	160	request.setHeader(b"X-Robots-Tag", "noindex, nofollow, noarchive, noimageindex")
156	161
157	162
158	163	# separators as defined in RFC2616. SP and HT are handled separately.

+1

-1

synapse/rest/media/v1/media_repository.py less more

65	65	def __init__(self, hs):
66	66	self.hs = hs
67	67	self.auth = hs.get_auth()
68		self.client = hs.get_http_client()
	68	self.client = hs.get_federation_http_client()
69	69	self.clock = hs.get_clock()
70	70	self.server_name = hs.hostname
71	71	self.store = hs.get_datastore()

+5

-1

synapse/rest/media/v1/preview_url_resource.py less more

675	675	logger.debug("No media removed from url cache")
676	676
677	677
678		def decode_and_calc_og(body, media_uri, request_encoding=None):
	678	def decode_and_calc_og(body, media_uri, request_encoding=None) -> Dict[str, str]:
	679	# If there's no body, nothing useful is going to be found.
	680	if not body:
	681	return {}
	682
679	683	from lxml import etree
680	684
681	685	try:

+6

-10

synapse/rest/media/v1/storage_provider.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15		import inspect
16	15	import logging
17	16	import os
18	17	import shutil

20	19
21	20	from synapse.config._base import Config
22	21	from synapse.logging.context import defer_to_thread, run_in_background
	22	from synapse.util.async_helpers import maybe_awaitable
23	23
24	24	from ._base import FileInfo, Responder
25	25	from .media_storage import FileResponder

90	90	if self.store_synchronous:
91	91	# store_file is supposed to return an Awaitable, but guard
92	92	# against improper implementations.
93		result = self.backend.store_file(path, file_info)
94		if inspect.isawaitable(result):
95		return await result
	93	return await maybe_awaitable(self.backend.store_file(path, file_info))
96	94	else:
97	95	# TODO: Handle errors.
98	96	async def store():
99	97	try:
100		result = self.backend.store_file(path, file_info)
101		if inspect.isawaitable(result):
102		return await result
	98	return await maybe_awaitable(
	99	self.backend.store_file(path, file_info)
	100	)
103	101	except Exception:
104	102	logger.exception("Error storing file")
105	103

109	107	async def fetch(self, path, file_info):
110	108	# store_file is supposed to return an Awaitable, but guard
111	109	# against improper implementations.
112		result = self.backend.fetch(path, file_info)
113		if inspect.isawaitable(result):
114		return await result
	110	return await maybe_awaitable(self.backend.fetch(path, file_info))
115	111
116	112
117	113	class FileStorageProviderBackend(StorageProvider):

+1

-1

synapse/rest/media/v1/upload_resource.py less more

43	43	requester = await self.auth.get_user_by_req(request)
44	44	# TODO: The checks here are a bit late. The content will have
45	45	# already been uploaded to a tmp file at this point
46		content_length = request.getHeader(b"Content-Length").decode("ascii")
	46	content_length = request.getHeader("Content-Length")
47	47	if content_length is None:
48	48	raise SynapseError(msg="Request must specify a Content-Length", code=400)
49	49	if int(content_length) > self.max_upload_size:

+88

-0

synapse/rest/synapse/client/pick_username.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	from typing import TYPE_CHECKING
	15
	16	import pkg_resources
	17
	18	from twisted.web.http import Request
	19	from twisted.web.resource import Resource
	20	from twisted.web.static import File
	21
	22	from synapse.api.errors import SynapseError
	23	from synapse.handlers.sso import USERNAME_MAPPING_SESSION_COOKIE_NAME
	24	from synapse.http.server import DirectServeHtmlResource, DirectServeJsonResource
	25	from synapse.http.servlet import parse_string
	26	from synapse.http.site import SynapseRequest
	27
	28	if TYPE_CHECKING:
	29	from synapse.server import HomeServer
	30
	31
	32	def pick_username_resource(hs: "HomeServer") -> Resource:
	33	"""Factory method to generate the username picker resource.
	34
	35	This resource gets mounted under /_synapse/client/pick_username. The top-level
	36	resource is just a File resource which serves up the static files in the resources
	37	"res" directory, but it has a couple of children:
	38
	39	* "submit", which does the mechanics of registering the new user, and redirects the
	40	browser back to the client URL
	41
	42	* "check": checks if a userid is free.
	43	"""
	44
	45	# XXX should we make this path customisable so that admins can restyle it?
	46	base_path = pkg_resources.resource_filename("synapse", "res/username_picker")
	47
	48	res = File(base_path)
	49	res.putChild(b"submit", SubmitResource(hs))
	50	res.putChild(b"check", AvailabilityCheckResource(hs))
	51
	52	return res
	53
	54
	55	class AvailabilityCheckResource(DirectServeJsonResource):
	56	def __init__(self, hs: "HomeServer"):
	57	super().__init__()
	58	self._sso_handler = hs.get_sso_handler()
	59
	60	async def _async_render_GET(self, request: Request):
	61	localpart = parse_string(request, "username", required=True)
	62
	63	session_id = request.getCookie(USERNAME_MAPPING_SESSION_COOKIE_NAME)
	64	if not session_id:
	65	raise SynapseError(code=400, msg="missing session_id")
	66
	67	is_available = await self._sso_handler.check_username_availability(
	68	localpart, session_id.decode("ascii", errors="replace")
	69	)
	70	return 200, {"available": is_available}
	71
	72
	73	class SubmitResource(DirectServeHtmlResource):
	74	def __init__(self, hs: "HomeServer"):
	75	super().__init__()
	76	self._sso_handler = hs.get_sso_handler()
	77
	78	async def _async_render_POST(self, request: SynapseRequest):
	79	localpart = parse_string(request, "username", required=True)
	80
	81	session_id = request.getCookie(USERNAME_MAPPING_SESSION_COOKIE_NAME)
	82	if not session_id:
	83	raise SynapseError(code=400, msg="missing session_id")
	84
	85	await self._sso_handler.handle_submit_username_request(
	86	request, localpart, session_id.decode("ascii", errors="replace")
	87	)

+31

-8

synapse/server.py less more

349	349
350	350	@cache_in_self
351	351	def get_simple_http_client(self) -> SimpleHttpClient:
	352	"""
	353	An HTTP client with no special configuration.
	354	"""
352	355	return SimpleHttpClient(self)
353	356
354	357	@cache_in_self
355	358	def get_proxied_http_client(self) -> SimpleHttpClient:
	359	"""
	360	An HTTP client that uses configured HTTP(S) proxies.
	361	"""
356	362	return SimpleHttpClient(
357	363	self,
358	364	http_proxy=os.getenvb(b"http_proxy"),

360	366	)
361	367
362	368	@cache_in_self
	369	def get_proxied_blacklisted_http_client(self) -> SimpleHttpClient:
	370	"""
	371	An HTTP client that uses configured HTTP(S) proxies and blacklists IPs
	372	based on the IP range blacklist/whitelist.
	373	"""
	374	return SimpleHttpClient(
	375	self,
	376	ip_whitelist=self.config.ip_range_whitelist,
	377	ip_blacklist=self.config.ip_range_blacklist,
	378	http_proxy=os.getenvb(b"http_proxy"),
	379	https_proxy=os.getenvb(b"HTTPS_PROXY"),
	380	)
	381
	382	@cache_in_self
	383	def get_federation_http_client(self) -> MatrixFederationHttpClient:
	384	"""
	385	An HTTP client for federation.
	386	"""
	387	tls_client_options_factory = context_factory.FederationPolicyForHTTPS(
	388	self.config
	389	)
	390	return MatrixFederationHttpClient(self, tls_client_options_factory)
	391
	392	@cache_in_self
363	393	def get_room_creation_handler(self) -> RoomCreationHandler:
364	394	return RoomCreationHandler(self)
365	395

512	542	@cache_in_self
513	543	def get_pusherpool(self) -> PusherPool:
514	544	return PusherPool(self)
515
516		@cache_in_self
517		def get_http_client(self) -> MatrixFederationHttpClient:
518		tls_client_options_factory = context_factory.FederationPolicyForHTTPS(
519		self.config
520		)
521		return MatrixFederationHttpClient(self, tls_client_options_factory)
522	545
523	546	@cache_in_self
524	547	def get_media_repository_resource(self) -> MediaRepositoryResource:

594	617	return StatsHandler(self)
595	618
596	619	@cache_in_self
597		def get_spam_checker(self):
	620	def get_spam_checker(self) -> SpamChecker:
598	621	return SpamChecker(self)
599	622
600	623	@cache_in_self

+2

-2

synapse/state/__init__.py less more

782	782	)
783	783
784	784	def get_auth_chain_difference(
785		self, state_sets: List[Set[str]]
	785	self, room_id: str, state_sets: List[Set[str]]
786	786	) -> Awaitable[Set[str]]:
787	787	"""Given sets of state events figure out the auth chain difference (as
788	788	per state res v2 algorithm).

795	795	An awaitable that resolves to a set of event IDs.
796	796	"""
797	797
798		return self.store.get_auth_chain_difference(state_sets)
	798	return self.store.get_auth_chain_difference(room_id, state_sets)

+89

-5

synapse/state/v2.py less more

37	37	from synapse.api.errors import AuthError
38	38	from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
39	39	from synapse.events import EventBase
40		from synapse.types import MutableStateMap, StateMap
	40	from synapse.types import Collection, MutableStateMap, StateMap
41	41	from synapse.util import Clock
42	42
43	43	logger = logging.getLogger(__name__)

96	96
97	97	# Also fetch all auth events that appear in only some of the state sets'
98	98	# auth chains.
99		auth_diff = await _get_auth_chain_difference(state_sets, event_map, state_res_store)
	99	auth_diff = await _get_auth_chain_difference(
	100	room_id, state_sets, event_map, state_res_store
	101	)
100	102
101	103	full_conflicted_set = set(
102	104	itertools.chain(

235	237
236	238
237	239	async def _get_auth_chain_difference(
	240	room_id: str,
238	241	state_sets: Sequence[StateMap[str]],
239	242	event_map: Dict[str, EventBase],
240	243	state_res_store: "synapse.state.StateResolutionStore",

251	254	Set of event IDs
252	255	"""
253	256
	257	# The `StateResolutionStore.get_auth_chain_difference` function assumes that
	258	# all events passed to it (and their auth chains) have been persisted
	259	# previously. This is not the case for any events in the `event_map`, and so
	260	# we need to manually handle those events.
	261	#
	262	# We do this by:
	263	# 1. calculating the auth chain difference for the state sets based on the
	264	# events in `event_map` alone
	265	# 2. replacing any events in the state_sets that are also in `event_map`
	266	# with their auth events (recursively), and then calling
	267	# `store.get_auth_chain_difference` as normal
	268	# 3. adding the results of 1 and 2 together.
	269
	270	# Map from event ID in `event_map` to their auth event IDs, and their auth
	271	# event IDs if they appear in the `event_map`. This is the intersection of
	272	# the event's auth chain with the events in the `event_map` plus their
	273	# auth event IDs.
	274	events_to_auth_chain = {} # type: Dict[str, Set[str]]
	275	for event in event_map.values():
	276	chain = {event.event_id}
	277	events_to_auth_chain[event.event_id] = chain
	278
	279	to_search = [event]
	280	while to_search:
	281	for auth_id in to_search.pop().auth_event_ids():
	282	chain.add(auth_id)
	283	auth_event = event_map.get(auth_id)
	284	if auth_event:
	285	to_search.append(auth_event)
	286
	287	# We now a) calculate the auth chain difference for the unpersisted events
	288	# and b) work out the state sets to pass to the store.
	289	#
	290	# Note: If the `event_map` is empty (which is the common case), we can do a
	291	# much simpler calculation.
	292	if event_map:
	293	# The list of state sets to pass to the store, where each state set is a set
	294	# of the event ids making up the state. This is similar to `state_sets`,
	295	# except that (a) we only have event ids, not the complete
	296	# ((type, state_key)->event_id) mappings; and (b) we have stripped out
	297	# unpersisted events and replaced them with the persisted events in
	298	# their auth chain.
	299	state_sets_ids = [] # type: List[Set[str]]
	300
	301	# For each state set, the unpersisted event IDs reachable (by their auth
	302	# chain) from the events in that set.
	303	unpersisted_set_ids = [] # type: List[Set[str]]
	304
	305	for state_set in state_sets:
	306	set_ids = set() # type: Set[str]
	307	state_sets_ids.append(set_ids)
	308
	309	unpersisted_ids = set() # type: Set[str]
	310	unpersisted_set_ids.append(unpersisted_ids)
	311
	312	for event_id in state_set.values():
	313	event_chain = events_to_auth_chain.get(event_id)
	314	if event_chain is not None:
	315	# We have an event in `event_map`. We add all the auth
	316	# events that it references (that aren't also in `event_map`).
	317	set_ids.update(e for e in event_chain if e not in event_map)
	318
	319	# We also add the full chain of unpersisted event IDs
	320	# referenced by this state set, so that we can work out the
	321	# auth chain difference of the unpersisted events.
	322	unpersisted_ids.update(e for e in event_chain if e in event_map)
	323	else:
	324	set_ids.add(event_id)
	325
	326	# The auth chain difference of the unpersisted events of the state sets
	327	# is calculated by taking the difference between the union and
	328	# intersections.
	329	union = unpersisted_set_ids[0].union(*unpersisted_set_ids[1:])
	330	intersection = unpersisted_set_ids[0].intersection(*unpersisted_set_ids[1:])
	331
	332	difference_from_event_map = union - intersection # type: Collection[str]
	333	else:
	334	difference_from_event_map = ()
	335	state_sets_ids = [set(state_set.values()) for state_set in state_sets]
	336
254	337	difference = await state_res_store.get_auth_chain_difference(
255		[set(state_set.values()) for state_set in state_sets]
256		)
	338	room_id, state_sets_ids
	339	)
	340	difference.update(difference_from_event_map)
257	341
258	342	return difference
259	343

573	657	# We do an iterative search, replacing `event with the power level in its
574	658	# auth events (if any)
575	659	while tmp_event:
576		depth = mainline_map.get(event.event_id)
	660	depth = mainline_map.get(tmp_event.event_id)
577	661	if depth is not None:
578	662	return depth
579	663

+7

-2

synapse/storage/__init__.py less more

26	26	data stores associated with them (e.g. the schema version tables), which are
27	27	stored in `synapse.storage.schema`.
28	28	"""
	29	from typing import TYPE_CHECKING
29	30
30	31	from synapse.storage.databases import Databases
31	32	from synapse.storage.databases.main import DataStore

33	34	from synapse.storage.purge_events import PurgeEventsStorage
34	35	from synapse.storage.state import StateGroupStorage
35	36
36		__all__ = ["DataStores", "DataStore"]
	37	if TYPE_CHECKING:
	38	from synapse.app.homeserver import HomeServer
	39
	40
	41	__all__ = ["Databases", "DataStore"]
37	42
38	43
39	44	class Storage:
40	45	"""The high level interfaces for talking to various storage layers.
41	46	"""
42	47
43		def __init__(self, hs, stores: Databases):
	48	def __init__(self, hs: "HomeServer", stores: Databases):
44	49	# We include the main data store here mainly so that we don't have to
45	50	# rewrite all the existing code to split it into high vs low level
46	51	# interfaces.

+25

-11

synapse/storage/_base.py less more

16	16	import logging
17	17	import random
18	18	from abc import ABCMeta
19		from typing import Any, Optional
	19	from typing import TYPE_CHECKING, Any, Iterable, Optional, Union
20	20
21	21	from synapse.storage.database import LoggingTransaction # noqa: F401
22	22	from synapse.storage.database import make_in_list_sql_clause # noqa: F401
23	23	from synapse.storage.database import DatabasePool
24		from synapse.types import Collection, get_domain_from_id
	24	from synapse.storage.types import Connection
	25	from synapse.types import Collection, StreamToken, get_domain_from_id
25	26	from synapse.util import json_decoder
	27
	28	if TYPE_CHECKING:
	29	from synapse.app.homeserver import HomeServer
26	30
27	31	logger = logging.getLogger(__name__)
28	32

35	39	per data store (and not one per physical database).
36	40	"""
37	41
38		def __init__(self, database: DatabasePool, db_conn, hs):
	42	def __init__(self, database: DatabasePool, db_conn: Connection, hs: "HomeServer"):
39	43	self.hs = hs
40	44	self._clock = hs.get_clock()
41	45	self.database_engine = database.engine
42	46	self.db_pool = database
43	47	self.rand = random.SystemRandom()
44	48
45		def process_replication_rows(self, stream_name, instance_name, token, rows):
	49	def process_replication_rows(
	50	self,
	51	stream_name: str,
	52	instance_name: str,
	53	token: StreamToken,
	54	rows: Iterable[Any],
	55	) -> None:
46	56	pass
47	57
48		def _invalidate_state_caches(self, room_id, members_changed):
	58	def _invalidate_state_caches(
	59	self, room_id: str, members_changed: Iterable[str]
	60	) -> None:
49	61	"""Invalidates caches that are based on the current state, but does
50	62	not stream invalidations down replication.
51	63
52	64	Args:
53		room_id (str): Room where state changed
54		members_changed (iterable[str]): The user_ids of members that have
55		changed
	65	room_id: Room where state changed
	66	members_changed: The user_ids of members that have changed
56	67	"""
57	68	for host in {get_domain_from_id(u) for u in members_changed}:
58	69	self._attempt_to_invalidate_cache("is_host_joined", (room_id, host))

63	74
64	75	def _attempt_to_invalidate_cache(
65	76	self, cache_name: str, key: Optional[Collection[Any]]
66		):
	77	) -> None:
67	78	"""Attempts to invalidate the cache of the given name, ignoring if the
68	79	cache doesn't exist. Mainly used for invalidating caches on workers,
69	80	where they may not have the cache.

87	98	cache.invalidate(tuple(key))
88	99
89	100
90		def db_to_json(db_content):
	101	def db_to_json(db_content: Union[memoryview, bytes, bytearray, str]) -> Any:
91	102	"""
92	103	Take some data from a database row and return a JSON-decoded object.
93	104
94	105	Args:
95		db_content (memoryview\|buffer\|bytes\|bytearray\|unicode)
	106	db_content: The JSON-encoded contents from the database.
	107
	108	Returns:
	109	The object decoded from JSON.
96	110	"""
97	111	# psycopg2 on Python 3 returns memoryview objects, which we need to
98	112	# cast to bytes to decode

+68

-51

synapse/storage/background_updates.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	import logging
16		from typing import Optional
	15	from typing import TYPE_CHECKING, Awaitable, Callable, Dict, Iterable, Optional
17	16
18	17	from synapse.metrics.background_process_metrics import run_as_background_process
	18	from synapse.storage.types import Connection
	19	from synapse.types import JsonDict
19	20	from synapse.util import json_encoder
20	21
21	22	from . import engines
	23
	24	if TYPE_CHECKING:
	25	from synapse.app.homeserver import HomeServer
	26	from synapse.storage.database import DatabasePool, LoggingTransaction
22	27
23	28	logger = logging.getLogger(__name__)
24	29

26	31	class BackgroundUpdatePerformance:
27	32	"""Tracks the how long a background update is taking to update its items"""
28	33
29		def __init__(self, name):
	34	def __init__(self, name: str):
30	35	self.name = name
31	36	self.total_item_count = 0
32		self.total_duration_ms = 0
33		self.avg_item_count = 0
34		self.avg_duration_ms = 0
35
36		def update(self, item_count, duration_ms):
	37	self.total_duration_ms = 0.0
	38	self.avg_item_count = 0.0
	39	self.avg_duration_ms = 0.0
	40
	41	def update(self, item_count: int, duration_ms: float) -> None:
37	42	"""Update the stats after doing an update"""
38	43	self.total_item_count += item_count
39	44	self.total_duration_ms += duration_ms

43	48	self.avg_item_count += 0.1 * (item_count - self.avg_item_count)
44	49	self.avg_duration_ms += 0.1 * (duration_ms - self.avg_duration_ms)
45	50
46		def average_items_per_ms(self):
	51	def average_items_per_ms(self) -> Optional[float]:
47	52	"""An estimate of how long it takes to do a single update.
48	53	Returns:
49	54	A duration in ms as a float

57	62	# changes in how long the update process takes.
58	63	return float(self.avg_item_count) / float(self.avg_duration_ms)
59	64
60		def total_items_per_ms(self):
	65	def total_items_per_ms(self) -> Optional[float]:
61	66	"""An estimate of how long it takes to do a single update.
62	67	Returns:
63	68	A duration in ms as a float

82	87	BACKGROUND_UPDATE_INTERVAL_MS = 1000
83	88	BACKGROUND_UPDATE_DURATION_MS = 100
84	89
85		def __init__(self, hs, database):
	90	def __init__(self, hs: "HomeServer", database: "DatabasePool"):
86	91	self._clock = hs.get_clock()
87	92	self.db_pool = database
88	93
89	94	# if a background update is currently running, its name.
90	95	self._current_background_update = None # type: Optional[str]
91	96
92		self._background_update_performance = {}
93		self._background_update_handlers = {}
	97	self._background_update_performance = (
	98	{}
	99	) # type: Dict[str, BackgroundUpdatePerformance]
	100	self._background_update_handlers = (
	101	{}
	102	) # type: Dict[str, Callable[[JsonDict, int], Awaitable[int]]]
94	103	self._all_done = False
95	104
96		def start_doing_background_updates(self):
	105	def start_doing_background_updates(self) -> None:
97	106	run_as_background_process("background_updates", self.run_background_updates)
98	107
99		async def run_background_updates(self, sleep=True):
	108	async def run_background_updates(self, sleep: bool = True) -> None:
100	109	logger.info("Starting background schema updates")
101	110	while True:
102	111	if sleep:

147	156
148	157	return False
149	158
150		async def has_completed_background_update(self, update_name) -> bool:
	159	async def has_completed_background_update(self, update_name: str) -> bool:
151	160	"""Check if the given background update has finished running.
152	161	"""
153	162	if self._all_done:

172	181	Returns once some amount of work is done.
173	182
174	183	Args:
175		desired_duration_ms(float): How long we want to spend
176		updating.
	184	desired_duration_ms: How long we want to spend updating.
177	185	Returns:
178	186	True if we have finished running all the background updates, otherwise False
179	187	"""

219	227	return False
220	228
221	229	async def _do_background_update(self, desired_duration_ms: float) -> int:
	230	assert self._current_background_update is not None
222	231	update_name = self._current_background_update
223	232	logger.info("Starting update batch on background update '%s'", update_name)
224	233

272	281
273	282	return len(self._background_update_performance)
274	283
275		def register_background_update_handler(self, update_name, update_handler):
	284	def register_background_update_handler(
	285	self,
	286	update_name: str,
	287	update_handler: Callable[[JsonDict, int], Awaitable[int]],
	288	):
276	289	"""Register a handler for doing a background update.
277	290
278	291	The handler should take two arguments:

286	299	The handler is responsible for updating the progress of the update.
287	300
288	301	Args:
289		update_name(str): The name of the update that this code handles.
290		update_handler(function): The function that does the update.
	302	update_name: The name of the update that this code handles.
	303	update_handler: The function that does the update.
291	304	"""
292	305	self._background_update_handlers[update_name] = update_handler
293	306
294		def register_noop_background_update(self, update_name):
	307	def register_noop_background_update(self, update_name: str) -> None:
295	308	"""Register a noop handler for a background update.
296	309
297	310	This is useful when we previously did a background update, but no

301	314	also be called to clear the update.
302	315
303	316	Args:
304		update_name (str): Name of update
305		"""
306
307		async def noop_update(progress, batch_size):
	317	update_name: Name of update
	318	"""
	319
	320	async def noop_update(progress: JsonDict, batch_size: int) -> int:
308	321	await self._end_background_update(update_name)
309	322	return 1
310	323

312	325
313	326	def register_background_index_update(
314	327	self,
315		update_name,
316		index_name,
317		table,
318		columns,
319		where_clause=None,
320		unique=False,
321		psql_only=False,
322		):
	328	update_name: str,
	329	index_name: str,
	330	table: str,
	331	columns: Iterable[str],
	332	where_clause: Optional[str] = None,
	333	unique: bool = False,
	334	psql_only: bool = False,
	335	) -> None:
323	336	"""Helper for store classes to do a background index addition
324	337
325	338	To use:

331	344	2. In the Store constructor, call this method
332	345
333	346	Args:
334		update_name (str): update_name to register for
335		index_name (str): name of index to add
336		table (str): table to add index to
337		columns (list[str]): columns/expressions to include in index
338		unique (bool): true to make a UNIQUE index
	347	update_name: update_name to register for
	348	index_name: name of index to add
	349	table: table to add index to
	350	columns: columns/expressions to include in index
	351	unique: true to make a UNIQUE index
339	352	psql_only: true to only create this index on psql databases (useful
340	353	for virtual sqlite tables)
341	354	"""
342	355
343		def create_index_psql(conn):
	356	def create_index_psql(conn: Connection) -> None:
344	357	conn.rollback()
345	358	# postgres insists on autocommit for the index
346		conn.set_session(autocommit=True)
	359	conn.set_session(autocommit=True) # type: ignore
347	360
348	361	try:
349	362	c = conn.cursor()

370	383	logger.debug("[SQL] %s", sql)
371	384	c.execute(sql)
372	385	finally:
373		conn.set_session(autocommit=False)
374
375		def create_index_sqlite(conn):
	386	conn.set_session(autocommit=False) # type: ignore
	387
	388	def create_index_sqlite(conn: Connection) -> None:
376	389	# Sqlite doesn't support concurrent creation of indexes.
377	390	#
378	391	# We don't use partial indices on SQLite as it wasn't introduced

398	411	c.execute(sql)
399	412
400	413	if isinstance(self.db_pool.engine, engines.PostgresEngine):
401		runner = create_index_psql
	414	runner = create_index_psql # type: Optional[Callable[[Connection], None]]
402	415	elif psql_only:
403	416	runner = None
404	417	else:

432	445	"background_updates", keyvalues={"update_name": update_name}
433	446	)
434	447
435		async def _background_update_progress(self, update_name: str, progress: dict):
	448	async def _background_update_progress(
	449	self, update_name: str, progress: dict
	450	) -> None:
436	451	"""Update the progress of a background update
437	452
438	453	Args:

440	455	progress: The progress of the update.
441	456	"""
442	457
443		return await self.db_pool.runInteraction(
	458	await self.db_pool.runInteraction(
444	459	"background_update_progress",
445	460	self._background_update_progress_txn,
446	461	update_name,
447	462	progress,
448	463	)
449	464
450		def _background_update_progress_txn(self, txn, update_name, progress):
	465	def _background_update_progress_txn(
	466	self, txn: "LoggingTransaction", update_name: str, progress: JsonDict
	467	) -> None:
451	468	"""Update the progress of a background update
452	469
453	470	Args:
454		txn(cursor): The transaction.
455		update_name(str): The name of the background update task
456		progress(dict): The progress of the update.
	471	txn: The transaction.
	472	update_name: The name of the background update task
	473	progress: The progress of the update.
457	474	"""
458	475
459	476	progress_json = json_encoder.encode(progress)

+4

-6

synapse/storage/databases/main/__init__.py less more

148	148	self._event_reports_id_gen = IdGenerator(db_conn, "event_reports", "id")
149	149	self._push_rule_id_gen = IdGenerator(db_conn, "push_rules", "id")
150	150	self._push_rules_enable_id_gen = IdGenerator(db_conn, "push_rules_enable", "id")
151		self._pushers_id_gen = StreamIdGenerator(
152		db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
153		)
154	151	self._group_updates_id_gen = StreamIdGenerator(
155	152	db_conn, "local_group_updates", "stream_id"
156	153	)

341	338	filters = []
342	339	args = [self.hs.config.server_name]
343	340
	341	# `name` is in database already in lower case
344	342	if name:
345		filters.append("(name LIKE ? OR displayname LIKE ?)")
346		args.extend(["@%" + name + "%:%", "%" + name + "%"])
	343	filters.append("(name LIKE ? OR LOWER(displayname) LIKE ?)")
	344	args.extend(["@%" + name.lower() + "%:%", "%" + name.lower() + "%"])
347	345	elif user_id:
348	346	filters.append("name LIKE ?")
349		args.extend(["%" + user_id + "%"])
	347	args.extend(["%" + user_id.lower() + "%"])
350	348
351	349	if not guests:
352	350	filters.append("is_guest = 0")

+5

-2

synapse/storage/databases/main/client_ips.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
16		from typing import Dict, Optional, Tuple
	16	from typing import Dict, List, Optional, Tuple, Union
17	17
18	18	from synapse.metrics.background_process_metrics import wrap_as_background_process
19	19	from synapse.storage._base import SQLBaseStore
20	20	from synapse.storage.database import DatabasePool, make_tuple_comparison_clause
	21	from synapse.types import UserID
21	22	from synapse.util.caches.lrucache import LruCache
22	23
23	24	logger = logging.getLogger(__name__)

545	546	}
546	547	return ret
547	548
548		async def get_user_ip_and_agents(self, user):
	549	async def get_user_ip_and_agents(
	550	self, user: UserID
	551	) -> List[Dict[str, Union[str, int]]]:
549	552	user_id = user.to_string()
550	553	results = {}
551	554

+32

-0

synapse/storage/databases/main/devices.py less more

55	55	self._clock.looping_call(
56	56	self._prune_old_outbound_device_pokes, 60 * 60 * 1000
57	57	)
	58
	59	async def count_devices_by_users(self, user_ids: Optional[List[str]] = None) -> int:
	60	"""Retrieve number of all devices of given users.
	61	Only returns number of devices that are not marked as hidden.
	62
	63	Args:
	64	user_ids: The IDs of the users which owns devices
	65	Returns:
	66	Number of devices of this users.
	67	"""
	68
	69	def count_devices_by_users_txn(txn, user_ids):
	70	sql = """
	71	SELECT count(*)
	72	FROM devices
	73	WHERE
	74	hidden = '0' AND
	75	"""
	76
	77	clause, args = make_in_list_sql_clause(
	78	txn.database_engine, "user_id", user_ids
	79	)
	80
	81	txn.execute(sql + clause, args)
	82	return txn.fetchone()[0]
	83
	84	if not user_ids:
	85	return 0
	86
	87	return await self.db_pool.runInteraction(
	88	"count_devices_by_users", count_devices_by_users_txn, user_ids
	89	)
58	90
59	91	async def get_device(self, user_id: str, device_id: str) -> Dict[str, Any]:
60	92	"""Retrieve a device. Only returns devices that are not marked as

+3

-1

synapse/storage/databases/main/event_federation.py less more

136	136
137	137	return list(results)
138	138
139		async def get_auth_chain_difference(self, state_sets: List[Set[str]]) -> Set[str]:
	139	async def get_auth_chain_difference(
	140	self, room_id: str, state_sets: List[Set[str]]
	141	) -> Set[str]:
140	142	"""Given sets of state events figure out the auth chain difference (as
141	143	per state res v2 algorithm).
142	144

+0

-10

synapse/storage/databases/main/event_push_actions.py less more

893	893	pa["actions"] = _deserialize_action(pa["actions"], pa["highlight"])
894	894	return push_actions
895	895
896		async def get_latest_push_action_stream_ordering(self):
897		def f(txn):
898		txn.execute("SELECT MAX(stream_ordering) FROM event_push_actions")
899		return txn.fetchone()
900
901		result = await self.db_pool.runInteraction(
902		"get_latest_push_action_stream_ordering", f
903		)
904		return result[0] or 0
905
906	896	def _remove_old_push_actions_before_txn(
907	897	self, txn, room_id, user_id, stream_ordering
908	898	):

+6

-6

synapse/storage/databases/main/keys.py less more

21	21
22	22	from synapse.storage._base import SQLBaseStore
23	23	from synapse.storage.keys import FetchKeyResult
	24	from synapse.storage.types import Cursor
24	25	from synapse.util.caches.descriptors import cached, cachedList
25	26	from synapse.util.iterutils import batch_iter
26	27

43	44	)
44	45	async def get_server_verify_keys(
45	46	self, server_name_and_key_ids: Iterable[Tuple[str, str]]
46		) -> Dict[Tuple[str, str], Optional[FetchKeyResult]]:
	47	) -> Dict[Tuple[str, str], FetchKeyResult]:
47	48	"""
48	49	Args:
49	50	server_name_and_key_ids:

55	56	"""
56	57	keys = {}
57	58
58		def _get_keys(txn, batch):
	59	def _get_keys(txn: Cursor, batch: Tuple[Tuple[str, str]]) -> None:
59	60	"""Processes a batch of keys to fetch, and adds the result to `keys`."""
60	61
61	62	# batch_iter always returns tuples so it's safe to do len(batch)

76	77	# `ts_valid_until_ms`.
77	78	ts_valid_until_ms = 0
78	79
79		res = FetchKeyResult(
	80	keys[(server_name, key_id)] = FetchKeyResult(
80	81	verify_key=decode_verify_key_bytes(key_id, bytes(key_bytes)),
81	82	valid_until_ts=ts_valid_until_ms,
82	83	)
83		keys[(server_name, key_id)] = res
84
85		def _txn(txn):
	84
	85	def _txn(txn: Cursor) -> Dict[Tuple[str, str], FetchKeyResult]:
86	86	for batch in batch_iter(server_name_and_key_ids, 50):
87	87	_get_keys(txn, batch)
88	88	return keys

+60

-39

synapse/storage/databases/main/pusher.py less more

14	14	# limitations under the License.
15	15
16	16	import logging
17		from typing import Iterable, Iterator, List, Tuple
	17	from typing import TYPE_CHECKING, Any, Dict, Iterable, Iterator, List, Optional, Tuple
18	18
19	19	from canonicaljson import encode_canonical_json
20	20
	21	from synapse.push import PusherConfig, ThrottleParams
21	22	from synapse.storage._base import SQLBaseStore, db_to_json
	23	from synapse.storage.database import DatabasePool
	24	from synapse.storage.types import Connection
	25	from synapse.storage.util.id_generators import StreamIdGenerator
	26	from synapse.types import JsonDict
22	27	from synapse.util.caches.descriptors import cached, cachedList
23	28
	29	if TYPE_CHECKING:
	30	from synapse.app.homeserver import HomeServer
	31
24	32	logger = logging.getLogger(__name__)
25	33
26	34
27	35	class PusherWorkerStore(SQLBaseStore):
28		def _decode_pushers_rows(self, rows: Iterable[dict]) -> Iterator[dict]:
	36	def __init__(self, database: DatabasePool, db_conn: Connection, hs: "HomeServer"):
	37	super().__init__(database, db_conn, hs)
	38	self._pushers_id_gen = StreamIdGenerator(
	39	db_conn, "pushers", "id", extra_tables=[("deleted_pushers", "stream_id")]
	40	)
	41
	42	def _decode_pushers_rows(self, rows: Iterable[dict]) -> Iterator[PusherConfig]:
29	43	"""JSON-decode the data in the rows returned from the `pushers` table
30	44
31	45	Drops any rows whose data cannot be decoded

43	57	)
44	58	continue
45	59
46		yield r
47
48		async def user_has_pusher(self, user_id):
	60	yield PusherConfig(**r)
	61
	62	async def user_has_pusher(self, user_id: str) -> bool:
49	63	ret = await self.db_pool.simple_select_one_onecol(
50	64	"pushers", {"user_name": user_id}, "id", allow_none=True
51	65	)
52	66	return ret is not None
53	67
54		def get_pushers_by_app_id_and_pushkey(self, app_id, pushkey):
55		return self.get_pushers_by({"app_id": app_id, "pushkey": pushkey})
56
57		def get_pushers_by_user_id(self, user_id):
58		return self.get_pushers_by({"user_name": user_id})
59
60		async def get_pushers_by(self, keyvalues):
	68	async def get_pushers_by_app_id_and_pushkey(
	69	self, app_id: str, pushkey: str
	70	) -> Iterator[PusherConfig]:
	71	return await self.get_pushers_by({"app_id": app_id, "pushkey": pushkey})
	72
	73	async def get_pushers_by_user_id(self, user_id: str) -> Iterator[PusherConfig]:
	74	return await self.get_pushers_by({"user_name": user_id})
	75
	76	async def get_pushers_by(self, keyvalues: Dict[str, Any]) -> Iterator[PusherConfig]:
61	77	ret = await self.db_pool.simple_select_list(
62	78	"pushers",
63	79	keyvalues,

82	98	)
83	99	return self._decode_pushers_rows(ret)
84	100
85		async def get_all_pushers(self):
	101	async def get_all_pushers(self) -> Iterator[PusherConfig]:
86	102	def get_pushers(txn):
87	103	txn.execute("SELECT * FROM pushers")
88	104	rows = self.db_pool.cursor_to_dict(txn)

158	174	)
159	175
160	176	@cached(num_args=1, max_entries=15000)
161		async def get_if_user_has_pusher(self, user_id):
	177	async def get_if_user_has_pusher(self, user_id: str):
162	178	# This only exists for the cachedList decorator
163	179	raise NotImplementedError()
164	180
165	181	@cachedList(
166	182	cached_method_name="get_if_user_has_pusher", list_name="user_ids", num_args=1,
167	183	)
168		async def get_if_users_have_pushers(self, user_ids):
	184	async def get_if_users_have_pushers(
	185	self, user_ids: Iterable[str]
	186	) -> Dict[str, bool]:
169	187	rows = await self.db_pool.simple_select_many_batch(
170	188	table="pushers",
171	189	column="user_name",

223	241	return bool(updated)
224	242
225	243	async def update_pusher_failing_since(
226		self, app_id, pushkey, user_id, failing_since
	244	self, app_id: str, pushkey: str, user_id: str, failing_since: Optional[int]
227	245	) -> None:
228	246	await self.db_pool.simple_update(
229	247	table="pushers",

232	250	desc="update_pusher_failing_since",
233	251	)
234	252
235		async def get_throttle_params_by_room(self, pusher_id):
	253	async def get_throttle_params_by_room(
	254	self, pusher_id: str
	255	) -> Dict[str, ThrottleParams]:
236	256	res = await self.db_pool.simple_select_list(
237	257	"pusher_throttle",
238	258	{"pusher": pusher_id},

242	262
243	263	params_by_room = {}
244	264	for row in res:
245		params_by_room[row["room_id"]] = {
246		"last_sent_ts": row["last_sent_ts"],
247		"throttle_ms": row["throttle_ms"],
248		}
	265	params_by_room[row["room_id"]] = ThrottleParams(
	266	row["last_sent_ts"], row["throttle_ms"],
	267	)
249	268
250	269	return params_by_room
251	270
252		async def set_throttle_params(self, pusher_id, room_id, params) -> None:
	271	async def set_throttle_params(
	272	self, pusher_id: str, room_id: str, params: ThrottleParams
	273	) -> None:
253	274	# no need to lock because `pusher_throttle` has a primary key on
254	275	# (pusher, room_id) so simple_upsert will retry
255	276	await self.db_pool.simple_upsert(
256	277	"pusher_throttle",
257	278	{"pusher": pusher_id, "room_id": room_id},
258		params,
	279	{"last_sent_ts": params.last_sent_ts, "throttle_ms": params.throttle_ms},
259	280	desc="set_throttle_params",
260	281	lock=False,
261	282	)
262	283
263	284
264	285	class PusherStore(PusherWorkerStore):
265		def get_pushers_stream_token(self):
	286	def get_pushers_stream_token(self) -> int:
266	287	return self._pushers_id_gen.get_current_token()
267	288
268	289	async def add_pusher(
269	290	self,
270		user_id,
271		access_token,
272		kind,
273		app_id,
274		app_display_name,
275		device_display_name,
276		pushkey,
277		pushkey_ts,
278		lang,
279		data,
280		last_stream_ordering,
281		profile_tag="",
	291	user_id: str,
	292	access_token: Optional[int],
	293	kind: str,
	294	app_id: str,
	295	app_display_name: str,
	296	device_display_name: str,
	297	pushkey: str,
	298	pushkey_ts: int,
	299	lang: Optional[str],
	300	data: Optional[JsonDict],
	301	last_stream_ordering: int,
	302	profile_tag: str = "",
282	303	) -> None:
283	304	async with self._pushers_id_gen.get_next() as stream_id:
284	305	# no need to lock because `pushers` has a unique key on

310	331	# invalidate, since we the user might not have had a pusher before
311	332	await self.db_pool.runInteraction(
312	333	"add_pusher",
313		self._invalidate_cache_and_stream,
	334	self._invalidate_cache_and_stream, # type: ignore
314	335	self.get_if_user_has_pusher,
315	336	(user_id,),
316	337	)
317	338
318	339	async def delete_pusher_by_app_id_pushkey_user_id(
319		self, app_id, pushkey, user_id
	340	self, app_id: str, pushkey: str, user_id: str
320	341	) -> None:
321	342	def delete_pusher_txn(txn, stream_id):
322		self._invalidate_cache_and_stream(
	343	self._invalidate_cache_and_stream( # type: ignore
323	344	txn, self.get_if_user_has_pusher, (user_id,)
324	345	)
325	346

+63

-0

synapse/storage/databases/main/registration.py less more

462	462	desc="get_user_by_external_id",
463	463	)
464	464
	465	async def get_external_ids_by_user(self, mxid: str) -> List[Tuple[str, str]]:
	466	"""Look up external ids for the given user
	467
	468	Args:
	469	mxid: the MXID to be looked up
	470
	471	Returns:
	472	Tuples of (auth_provider, external_id)
	473	"""
	474	res = await self.db_pool.simple_select_list(
	475	table="user_external_ids",
	476	keyvalues={"user_id": mxid},
	477	retcols=("auth_provider", "external_id"),
	478	desc="get_external_ids_by_user",
	479	)
	480	return [(r["auth_provider"], r["external_id"]) for r in res]
	481
465	482	async def count_all_users(self):
466	483	"""Counts all users registered on the homeserver."""
467	484

925	942	desc="del_user_pending_deactivation",
926	943	)
927	944
	945	async def get_access_token_last_validated(self, token_id: int) -> int:
	946	"""Retrieves the time (in milliseconds) of the last validation of an access token.
	947
	948	Args:
	949	token_id: The ID of the access token to update.
	950	Raises:
	951	StoreError if the access token was not found.
	952
	953	Returns:
	954	The last validation time.
	955	"""
	956	result = await self.db_pool.simple_select_one_onecol(
	957	"access_tokens", {"id": token_id}, "last_validated"
	958	)
	959
	960	# If this token has not been validated (since starting to track this),
	961	# return 0 instead of None.
	962	return result or 0
	963
	964	async def update_access_token_last_validated(self, token_id: int) -> None:
	965	"""Updates the last time an access token was validated.
	966
	967	Args:
	968	token_id: The ID of the access token to update.
	969	Raises:
	970	StoreError if there was a problem updating this.
	971	"""
	972	now = self._clock.time_msec()
	973
	974	await self.db_pool.simple_update_one(
	975	"access_tokens",
	976	{"id": token_id},
	977	{"last_validated": now},
	978	desc="update_access_token_last_validated",
	979	)
	980
928	981
929	982	class RegistrationBackgroundUpdateStore(RegistrationWorkerStore):
930	983	def __init__(self, database: DatabasePool, db_conn: Connection, hs: "HomeServer"):

960	1013
961	1014	self.db_pool.updates.register_background_update_handler(
962	1015	"users_set_deactivated_flag", self._background_update_set_deactivated_flag
	1016	)
	1017
	1018	self.db_pool.updates.register_background_index_update(
	1019	"user_external_ids_user_id_idx",
	1020	index_name="user_external_ids_user_id_idx",
	1021	table="user_external_ids",
	1022	columns=["user_id"],
	1023	unique=False,
963	1024	)
964	1025
965	1026	async def _background_update_set_deactivated_flag(self, progress, batch_size):

1124	1185	The token ID
1125	1186	"""
1126	1187	next_id = self._access_tokens_id_gen.get_next()
	1188	now = self._clock.time_msec()
1127	1189
1128	1190	await self.db_pool.simple_insert(
1129	1191	"access_tokens",

1134	1196	"device_id": device_id,
1135	1197	"valid_until_ms": valid_until_ms,
1136	1198	"puppets_user_id": puppets_user_id,
	1199	"last_validated": now,
1137	1200	},
1138	1201	desc="add_access_token_to_user",
1139	1202	)

+2

-2

synapse/storage/databases/main/room.py less more

378	378	# Filter room names by a string
379	379	where_statement = ""
380	380	if search_term:
381		where_statement = "WHERE state.name LIKE ?"
	381	where_statement = "WHERE LOWER(state.name) LIKE ?"
382	382
383	383	# Our postgres db driver converts ? -> %s in SQL strings as that's the
384	384	# placeholder for postgres.
385	385	# HOWEVER, if you put a % into your SQL then everything goes wibbly.
386	386	# To get around this, we're going to surround search_term with %'s
387	387	# before giving it to the database in python instead
388		search_term = "%" + search_term + "%"
	388	search_term = "%" + search_term.lower() + "%"
389	389
390	390	# Set ordering
391	391	if RoomSortOrder(order_by) == RoomSortOrder.SIZE:

+17

-0

synapse/storage/databases/main/schema/delta/58/25user_external_ids_user_id_idx.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	INSERT INTO background_updates (ordering, update_name, progress_json) VALUES
	16	(5825, 'user_external_ids_user_id_idx', '{}');

+18

-0

synapse/storage/databases/main/schema/delta/58/26access_token_last_validated.sql less more

	0	/* Copyright 2020 The Matrix.org Foundation C.I.C
	1	*
	2	* Licensed under the Apache License, Version 2.0 (the "License");
	3	* you may not use this file except in compliance with the License.
	4	* You may obtain a copy of the License at
	5	*
	6	* http://www.apache.org/licenses/LICENSE-2.0
	7	*
	8	* Unless required by applicable law or agreed to in writing, software
	9	* distributed under the License is distributed on an "AS IS" BASIS,
	10	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	* See the License for the specific language governing permissions and
	12	* limitations under the License.
	13	*/
	14
	15	-- The last time this access token was "validated" (i.e. logged in or succeeded
	16	-- at user-interactive authentication).
	17	ALTER TABLE access_tokens ADD COLUMN last_validated BIGINT;

+18

-0

synapse/storage/databases/main/schema/delta/58/27local_invites.sql less more

	0	/*
	1	* Copyright 2020 The Matrix.org Foundation C.I.C.
	2	*
	3	* Licensed under the Apache License, Version 2.0 (the "License");
	4	* you may not use this file except in compliance with the License.
	5	* You may obtain a copy of the License at
	6	*
	7	* http://www.apache.org/licenses/LICENSE-2.0
	8	*
	9	* Unless required by applicable law or agreed to in writing, software
	10	* distributed under the License is distributed on an "AS IS" BASIS,
	11	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	* See the License for the specific language governing permissions and
	13	* limitations under the License.
	14	*/
	15
	16	-- This is unused since Synapse v1.17.0.
	17	DROP TABLE local_invites;

+17

-14

synapse/storage/databases/main/user_directory.py less more

16	16	import re
17	17	from typing import Any, Dict, Iterable, Optional, Set, Tuple
18	18
19		from synapse.api.constants import EventTypes, JoinRules
	19	from synapse.api.constants import EventTypes, HistoryVisibility, JoinRules
20	20	from synapse.storage.database import DatabasePool
21	21	from synapse.storage.databases.main.state import StateFilter
22	22	from synapse.storage.databases.main.state_deltas import StateDeltasStore

359	359	if hist_vis_id:
360	360	hist_vis_ev = await self.get_event(hist_vis_id, allow_none=True)
361	361	if hist_vis_ev:
362		if hist_vis_ev.content.get("history_visibility") == "world_readable":
	362	if (
	363	hist_vis_ev.content.get("history_visibility")
	364	== HistoryVisibility.WORLD_READABLE
	365	):
363	366	return True
364	367
365	368	return False

392	395	sql = """
393	396	INSERT INTO user_directory_search(user_id, vector)
394	397	VALUES (?,
395		setweight(to_tsvector('english', ?), 'A')
396		\|\| setweight(to_tsvector('english', ?), 'D')
397		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	398	setweight(to_tsvector('simple', ?), 'A')
	399	\|\| setweight(to_tsvector('simple', ?), 'D')
	400	\|\| setweight(to_tsvector('simple', COALESCE(?, '')), 'B')
398	401	) ON CONFLICT (user_id) DO UPDATE SET vector=EXCLUDED.vector
399	402	"""
400	403	txn.execute(

414	417	sql = """
415	418	INSERT INTO user_directory_search(user_id, vector)
416	419	VALUES (?,
417		setweight(to_tsvector('english', ?), 'A')
418		\|\| setweight(to_tsvector('english', ?), 'D')
419		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	420	setweight(to_tsvector('simple', ?), 'A')
	421	\|\| setweight(to_tsvector('simple', ?), 'D')
	422	\|\| setweight(to_tsvector('simple', COALESCE(?, '')), 'B')
420	423	)
421	424	"""
422	425	txn.execute(

431	434	elif new_entry is False:
432	435	sql = """
433	436	UPDATE user_directory_search
434		SET vector = setweight(to_tsvector('english', ?), 'A')
435		\|\| setweight(to_tsvector('english', ?), 'D')
436		\|\| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	437	SET vector = setweight(to_tsvector('simple', ?), 'A')
	438	\|\| setweight(to_tsvector('simple', ?), 'D')
	439	\|\| setweight(to_tsvector('simple', COALESCE(?, '')), 'B')
437	440	WHERE user_id = ?
438	441	"""
439	442	txn.execute(

760	763	INNER JOIN user_directory AS d USING (user_id)
761	764	WHERE
762	765	%s
763		AND vector @@ to_tsquery('english', ?)
	766	AND vector @@ to_tsquery('simple', ?)
764	767	ORDER BY
765	768	(CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END)
766	769	* (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END)

769	772	3 * ts_rank_cd(
770	773	'{0.1, 0.1, 0.9, 1.0}',
771	774	vector,
772		to_tsquery('english', ?),
	775	to_tsquery('simple', ?),
773	776	8
774	777	)
775	778	+ ts_rank_cd(
776	779	'{0.1, 0.1, 0.9, 1.0}',
777	780	vector,
778		to_tsquery('english', ?),
	781	to_tsquery('simple', ?),
779	782	8
780	783	)
781	784	)

+3

-2

synapse/storage/keys.py less more

16	16	import logging
17	17
18	18	import attr
	19	from signedjson.types import VerifyKey
19	20
20	21	logger = logging.getLogger(__name__)
21	22
22	23
23	24	@attr.s(slots=True, frozen=True)
24	25	class FetchKeyResult:
25		verify_key = attr.ib() # VerifyKey: the key itself
26		valid_until_ts = attr.ib() # int: how long we can use this key for
	26	verify_key = attr.ib(type=VerifyKey) # the key itself
	27	valid_until_ts = attr.ib(type=int) # how long we can use this key for

+184

-16

synapse/storage/persist_events.py less more

30	30	from synapse.metrics.background_process_metrics import run_as_background_process
31	31	from synapse.storage.databases import Databases
32	32	from synapse.storage.databases.main.events import DeltaState
33		from synapse.types import Collection, PersistedEventPosition, RoomStreamToken, StateMap
	33	from synapse.storage.databases.main.events_worker import EventRedactBehaviour
	34	from synapse.types import (
	35	Collection,
	36	PersistedEventPosition,
	37	RoomStreamToken,
	38	StateMap,
	39	get_domain_from_id,
	40	)
34	41	from synapse.util.async_helpers import ObservableDeferred
35	42	from synapse.util.metrics import Measure
36	43

65	72	"synapse_storage_events_stale_forward_extremities_persisted",
66	73	"Number of unchanged forward extremities for each new event",
67	74	buckets=(0, 1, 2, 3, 5, 7, 10, 15, 20, 50, 100, 200, 500, "+Inf"),
	75	)
	76
	77	state_resolutions_during_persistence = Counter(
	78	"synapse_storage_events_state_resolutions_during_persistence",
	79	"Number of times we had to do state res to calculate new current state",
	80	)
	81
	82	potential_times_prune_extremities = Counter(
	83	"synapse_storage_events_potential_times_prune_extremities",
	84	"Number of times we might be able to prune extremities",
	85	)
	86
	87	times_pruned_extremities = Counter(
	88	"synapse_storage_events_times_pruned_extremities",
	89	"Number of times we were actually be able to prune extremities",
68	90	)
69	91
70	92

453	475	latest_event_ids,
454	476	new_latest_event_ids,
455	477	)
456		current_state, delta_ids = res
	478	current_state, delta_ids, new_latest_event_ids = res
	479
	480	# there should always be at least one forward extremity.
	481	# (except during the initial persistence of the send_join
	482	# results, in which case there will be no existing
	483	# extremities, so we'll `continue` above and skip this bit.)
	484	assert new_latest_event_ids, "No forward extremities left!"
	485
	486	new_forward_extremeties[room_id] = new_latest_event_ids
457	487
458	488	# If either are not None then there has been a change,
459	489	# and we need to work out the delta (or use that

572	602	self,
573	603	room_id: str,
574	604	events_context: List[Tuple[EventBase, EventContext]],
575		old_latest_event_ids: Iterable[str],
576		new_latest_event_ids: Iterable[str],
577		) -> Tuple[Optional[StateMap[str]], Optional[StateMap[str]]]:
	605	old_latest_event_ids: Set[str],
	606	new_latest_event_ids: Set[str],
	607	) -> Tuple[Optional[StateMap[str]], Optional[StateMap[str]], Set[str]]:
578	608	"""Calculate the current state dict after adding some new events to
579	609	a room
580	610
581	611	Args:
582		room_id (str):
	612	room_id:
583	613	room to which the events are being added. Used for logging etc
584	614
585		events_context (list[(EventBase, EventContext)]):
	615	events_context:
586	616	events and contexts which are being added to the room
587	617
588		old_latest_event_ids (iterable[str]):
	618	old_latest_event_ids:
589	619	the old forward extremities for the room.
590	620
591		new_latest_event_ids (iterable[str]):
	621	new_latest_event_ids :
592	622	the new forward extremities for the room.
593	623
594	624	Returns:
595		Returns a tuple of two state maps, the first being the full new current
596		state and the second being the delta to the existing current state.
597		If both are None then there has been no change.
	625	Returns a tuple of two state maps and a set of new forward
	626	extremities.
	627
	628	The first state map is the full new current state and the second
	629	is the delta to the existing current state. If both are None then
	630	there has been no change.
	631
	632	The function may prune some old entries from the set of new
	633	forward extremities if it's safe to do so.
598	634
599	635	If there has been a change then we only return the delta if its
600	636	already been calculated. Conversely if we do know the delta then

671	707	# If they old and new groups are the same then we don't need to do
672	708	# anything.
673	709	if old_state_groups == new_state_groups:
674		return None, None
	710	return None, None, new_latest_event_ids
675	711
676	712	if len(new_state_groups) == 1 and len(old_state_groups) == 1:
677	713	# If we're going from one state group to another, lets check if

688	724	# the current state in memory then lets also return that,
689	725	# but it doesn't matter if we don't.
690	726	new_state = state_groups_map.get(new_state_group)
691		return new_state, delta_ids
	727	return new_state, delta_ids, new_latest_event_ids
692	728
693	729	# Now that we have calculated new_state_groups we need to get
694	730	# their state IDs so we can resolve to a single state set.

700	736	if len(new_state_groups) == 1:
701	737	# If there is only one state group, then we know what the current
702	738	# state is.
703		return state_groups_map[new_state_groups.pop()], None
	739	return state_groups_map[new_state_groups.pop()], None, new_latest_event_ids
704	740
705	741	# Ok, we need to defer to the state handler to resolve our state sets.
706	742

733	769	state_res_store=StateResolutionStore(self.main_store),
734	770	)
735	771
736		return res.state, None
	772	state_resolutions_during_persistence.inc()
	773
	774	# If the returned state matches the state group of one of the new
	775	# forward extremities then we check if we are able to prune some state
	776	# extremities.
	777	if res.state_group and res.state_group in new_state_groups:
	778	new_latest_event_ids = await self._prune_extremities(
	779	room_id,
	780	new_latest_event_ids,
	781	res.state_group,
	782	event_id_to_state_group,
	783	events_context,
	784	)
	785
	786	return res.state, None, new_latest_event_ids
	787
	788	async def _prune_extremities(
	789	self,
	790	room_id: str,
	791	new_latest_event_ids: Set[str],
	792	resolved_state_group: int,
	793	event_id_to_state_group: Dict[str, int],
	794	events_context: List[Tuple[EventBase, EventContext]],
	795	) -> Set[str]:
	796	"""See if we can prune any of the extremities after calculating the
	797	resolved state.
	798	"""
	799	potential_times_prune_extremities.inc()
	800
	801	# We keep all the extremities that have the same state group, and
	802	# see if we can drop the others.
	803	new_new_extrems = {
	804	e
	805	for e in new_latest_event_ids
	806	if event_id_to_state_group[e] == resolved_state_group
	807	}
	808
	809	dropped_extrems = set(new_latest_event_ids) - new_new_extrems
	810
	811	logger.debug("Might drop extremities: %s", dropped_extrems)
	812
	813	# We only drop events from the extremities list if:
	814	# 1. we're not currently persisting them;
	815	# 2. they're not our own events (or are dummy events); and
	816	# 3. they're either:
	817	# 1. over N hours old and more than N events ago (we use depth to
	818	# calculate); or
	819	# 2. we are persisting an event from the same domain and more than
	820	# M events ago.
	821	#
	822	# The idea is that we don't want to drop events that are "legitimate"
	823	# extremities (that we would want to include as prev events), only
	824	# "stuck" extremities that are e.g. due to a gap in the graph.
	825	#
	826	# Note that we either drop all of them or none of them. If we only drop
	827	# some of the events we don't know if state res would come to the same
	828	# conclusion.
	829
	830	for ev, _ in events_context:
	831	if ev.event_id in dropped_extrems:
	832	logger.debug(
	833	"Not dropping extremities: %s is being persisted", ev.event_id
	834	)
	835	return new_latest_event_ids
	836
	837	dropped_events = await self.main_store.get_events(
	838	dropped_extrems,
	839	allow_rejected=True,
	840	redact_behaviour=EventRedactBehaviour.AS_IS,
	841	)
	842
	843	new_senders = {get_domain_from_id(e.sender) for e, _ in events_context}
	844
	845	one_day_ago = self._clock.time_msec() - 24 * 60 * 60 * 1000
	846	current_depth = max(e.depth for e, _ in events_context)
	847	for event in dropped_events.values():
	848	# If the event is a local dummy event then we should check it
	849	# doesn't reference any local events, as we want to reference those
	850	# if we send any new events.
	851	#
	852	# Note we do this recursively to handle the case where a dummy event
	853	# references a dummy event that only references remote events.
	854	#
	855	# Ideally we'd figure out a way of still being able to drop old
	856	# dummy events that reference local events, but this is good enough
	857	# as a first cut.
	858	events_to_check = [event]
	859	while events_to_check:
	860	new_events = set()
	861	for event_to_check in events_to_check:
	862	if self.is_mine_id(event_to_check.sender):
	863	if event_to_check.type != EventTypes.Dummy:
	864	logger.debug("Not dropping own event")
	865	return new_latest_event_ids
	866	new_events.update(event_to_check.prev_event_ids())
	867
	868	prev_events = await self.main_store.get_events(
	869	new_events,
	870	allow_rejected=True,
	871	redact_behaviour=EventRedactBehaviour.AS_IS,
	872	)
	873	events_to_check = prev_events.values()
	874
	875	if (
	876	event.origin_server_ts < one_day_ago
	877	and event.depth < current_depth - 100
	878	):
	879	continue
	880
	881	# We can be less conservative about dropping extremities from the
	882	# same domain, though we do want to wait a little bit (otherwise
	883	# we'll immediately remove all extremities from a given server).
	884	if (
	885	get_domain_from_id(event.sender) in new_senders
	886	and event.depth < current_depth - 20
	887	):
	888	continue
	889
	890	logger.debug(
	891	"Not dropping as too new and not in new_senders: %s", new_senders,
	892	)
	893
	894	return new_latest_event_ids
	895
	896	times_pruned_extremities.inc()
	897
	898	logger.info(
	899	"Pruning forward extremities in room %s: from %s -> %s",
	900	room_id,
	901	new_latest_event_ids,
	902	new_new_extrems,
	903	)
	904	return new_new_extrems
737	905
738	906	async def _calculate_state_delta(
739	907	self, room_id: str, current_state: StateMap[str]

+56

-48

synapse/storage/prepare_database.py less more

17	17	import os
18	18	import re
19	19	from collections import Counter
20		from typing import Optional, TextIO
	20	from typing import Generator, Iterable, List, Optional, TextIO, Tuple
21	21
22	22	import attr
	23	from typing_extensions import Counter as CounterType
23	24
24	25	from synapse.config.homeserver import HomeServerConfig
25	26	from synapse.storage.database import LoggingDatabaseConnection

69	70	db_conn: LoggingDatabaseConnection,
70	71	database_engine: BaseDatabaseEngine,
71	72	config: Optional[HomeServerConfig],
72		databases: Collection[str] = ["main", "state"],
	73	databases: Collection[str] = ("main", "state"),
73	74	):
74	75	"""Prepares a physical database for usage. Will either create all necessary tables
75	76	or upgrade from an older schema version.

154	155	raise
155	156
156	157
157		def _setup_new_database(cur, database_engine, databases):
	158	def _setup_new_database(
	159	cur: Cursor, database_engine: BaseDatabaseEngine, databases: Collection[str]
	160	) -> None:
158	161	"""Sets up the physical database by finding a base set of "full schemas" and
159	162	then applying any necessary deltas, including schemas from the given data
160	163	stores.

187	190	folder as well those in the data stores specified.
188	191
189	192	Args:
190		cur (Cursor): a database cursor
191		database_engine (DatabaseEngine)
192		databases (list[str]): The names of the databases to instantiate
193		on the given physical database.
	193	cur: a database cursor
	194	database_engine
	195	databases: The names of the databases to instantiate on the given physical database.
194	196	"""
195	197
196	198	# We're about to set up a brand new database so we check that its

198	200	database_engine.check_new_database(cur)
199	201
200	202	current_dir = os.path.join(dir_path, "schema", "full_schemas")
201		directory_entries = os.listdir(current_dir)
202	203
203	204	# First we find the highest full schema version we have
204	205	valid_versions = []
205	206
206		for filename in directory_entries:
	207	for filename in os.listdir(current_dir):
207	208	try:
208	209	ver = int(filename)
209	210	except ValueError:

236	237	for database in databases
237	238	)
238	239
239		directory_entries = []
	240	directory_entries = [] # type: List[_DirectoryListing]
240	241	for directory in directories:
241	242	directory_entries.extend(
242	243	_DirectoryListing(file_name, os.path.join(directory, file_name))

274	275
275	276
276	277	def _upgrade_existing_database(
277		cur,
278		current_version,
279		applied_delta_files,
280		upgraded,
281		database_engine,
282		config,
283		databases,
284		is_empty=False,
285		):
	278	cur: Cursor,
	279	current_version: int,
	280	applied_delta_files: List[str],
	281	upgraded: bool,
	282	database_engine: BaseDatabaseEngine,
	283	config: Optional[HomeServerConfig],
	284	databases: Collection[str],
	285	is_empty: bool = False,
	286	) -> None:
286	287	"""Upgrades an existing physical database.
287	288
288	289	Delta files can either be SQL stored in *.sql files, or python modules

322	323	for a version before applying those in the next version.
323	324
324	325	Args:
325		cur (Cursor)
326		current_version (int): The current version of the schema.
327		applied_delta_files (list): A list of deltas that have already been
328		applied.
329		upgraded (bool): Whether the current version was generated by having
	326	cur
	327	current_version: The current version of the schema.
	328	applied_delta_files: A list of deltas that have already been applied.
	329	upgraded: Whether the current version was generated by having
330	330	applied deltas or from full schema file. If `True` the function
331	331	will never apply delta files for the given `current_version`, since
332	332	the current_version wasn't generated by applying those delta files.
333		database_engine (DatabaseEngine)
334		config (synapse.config.homeserver.HomeServerConfig\|None):
	333	database_engine
	334	config:
335	335	None if we are initialising a blank database, otherwise the application
336	336	config
337		databases (list[str]): The names of the databases to instantiate
	337	databases: The names of the databases to instantiate
338	338	on the given physical database.
339		is_empty (bool): Is this a blank database? I.e. do we need to run the
	339	is_empty: Is this a blank database? I.e. do we need to run the
340	340	upgrade portions of the delta scripts.
341	341	"""
342	342	if is_empty:

357	357	if not is_empty and "main" in databases:
358	358	from synapse.storage.databases.main import check_database_before_upgrade
359	359
	360	assert config is not None
360	361	check_database_before_upgrade(cur, database_engine, config)
361	362
362	363	start_ver = current_version

387	388	)
388	389
389	390	# Used to check if we have any duplicate file names
390		file_name_counter = Counter()
	391	file_name_counter = Counter() # type: CounterType[str]
391	392
392	393	# Now find which directories have anything of interest.
393		directory_entries = []
	394	directory_entries = [] # type: List[_DirectoryListing]
394	395	for directory in directories:
395	396	logger.debug("Looking for schema deltas in %s", directory)
396	397	try:

444	445
445	446	module_name = "synapse.storage.v%d_%s" % (v, root_name)
446	447	with open(absolute_path) as python_file:
447		module = imp.load_source(module_name, absolute_path, python_file)
	448	module = imp.load_source(module_name, absolute_path, python_file) # type: ignore
448	449	logger.info("Running script %s", relative_path)
449		module.run_create(cur, database_engine)
	450	module.run_create(cur, database_engine) # type: ignore
450	451	if not is_empty:
451		module.run_upgrade(cur, database_engine, config=config)
	452	module.run_upgrade(cur, database_engine, config=config) # type: ignore
452	453	elif ext == ".pyc" or file_name == "__pycache__":
453	454	# Sometimes .pyc files turn up anyway even though we've
454	455	# disabled their generation; e.g. from distribution package

496	497	logger.info("Schema now up to date")
497	498
498	499
499		def _apply_module_schemas(txn, database_engine, config):
	500	def _apply_module_schemas(
	501	txn: Cursor, database_engine: BaseDatabaseEngine, config: HomeServerConfig
	502	) -> None:
500	503	"""Apply the module schemas for the dynamic modules, if any
501	504
502	505	Args:
503	506	cur: database cursor
504		database_engine: synapse database engine class
505		config (synapse.config.homeserver.HomeServerConfig):
506		application config
	507	database_engine:
	508	config: application config
507	509	"""
508	510	for (mod, _config) in config.password_providers:
509	511	if not hasattr(mod, "get_db_schema_files"):

514	516	)
515	517
516	518
517		def _apply_module_schema_files(cur, database_engine, modname, names_and_streams):
	519	def _apply_module_schema_files(
	520	cur: Cursor,
	521	database_engine: BaseDatabaseEngine,
	522	modname: str,
	523	names_and_streams: Iterable[Tuple[str, TextIO]],
	524	) -> None:
518	525	"""Apply the module schemas for a single module
519	526
520	527	Args:
521	528	cur: database cursor
522	529	database_engine: synapse database engine class
523		modname (str): fully qualified name of the module
524		names_and_streams (Iterable[(str, file)]): the names and streams of
525		schemas to be applied
	530	modname: fully qualified name of the module
	531	names_and_streams: the names and streams of schemas to be applied
526	532	"""
527	533	cur.execute(
528	534	"SELECT file FROM applied_module_schemas WHERE module_name = ?", (modname,),

548	554	)
549	555
550	556
551		def get_statements(f):
	557	def get_statements(f: Iterable[str]) -> Generator[str, None, None]:
552	558	statement_buffer = ""
553	559	in_comment = False # If we're in a /* ... */ style comment
554	560

593	599	statement_buffer = statements[-1].strip()
594	600
595	601
596		def executescript(txn, schema_path):
	602	def executescript(txn: Cursor, schema_path: str) -> None:
597	603	with open(schema_path, "r") as f:
598	604	execute_statements_from_stream(txn, f)
599	605
600	606
601		def execute_statements_from_stream(cur: Cursor, f: TextIO):
	607	def execute_statements_from_stream(cur: Cursor, f: TextIO) -> None:
602	608	for statement in get_statements(f):
603	609	cur.execute(statement)
604	610
605	611
606		def _get_or_create_schema_state(txn, database_engine):
	612	def _get_or_create_schema_state(
	613	txn: Cursor, database_engine: BaseDatabaseEngine
	614	) -> Optional[Tuple[int, List[str], bool]]:
607	615	# Bluntly try creating the schema_version tables.
608	616	schema_path = os.path.join(dir_path, "schema", "schema_version.sql")
609	617	executescript(txn, schema_path)

611	619	txn.execute("SELECT version, upgraded FROM schema_version")
612	620	row = txn.fetchone()
613	621	current_version = int(row[0]) if row else None
614		upgraded = bool(row[1]) if row else None
615	622
616	623	if current_version:
617	624	txn.execute(

619	626	(current_version,),
620	627	)
621	628	applied_deltas = [d for d, in txn]
	629	upgraded = bool(row[1])
622	630	return current_version, applied_deltas, upgraded
623	631
624	632	return None

633	641	`file_name` attr is kept first.
634	642	"""
635	643
636		file_name = attr.ib()
637		absolute_path = attr.ib()
	644	file_name = attr.ib(type=str)
	645	absolute_path = attr.ib(type=str)

+8

-3

synapse/storage/purge_events.py less more

14	14
15	15	import itertools
16	16	import logging
17		from typing import Set
	17	from typing import TYPE_CHECKING, Set
	18
	19	from synapse.storage.databases import Databases
	20
	21	if TYPE_CHECKING:
	22	from synapse.app.homeserver import HomeServer
18	23
19	24	logger = logging.getLogger(__name__)
20	25

23	28	"""High level interface for purging rooms and event history.
24	29	"""
25	30
26		def __init__(self, hs, stores):
	31	def __init__(self, hs: "HomeServer", stores: Databases):
27	32	self.stores = stores
28	33
29		async def purge_room(self, room_id: str):
	34	async def purge_room(self, room_id: str) -> None:
30	35	"""Deletes all record of a room
31	36	"""
32	37

+23

-21

synapse/storage/relations.py less more

13	13	# limitations under the License.
14	14
15	15	import logging
	16	from typing import Any, Dict, List, Optional, Tuple
16	17
17	18	import attr
18	19
19	20	from synapse.api.errors import SynapseError
	21	from synapse.types import JsonDict
20	22
21	23	logger = logging.getLogger(__name__)
22	24

26	28	"""Returned by relation pagination APIs.
27	29
28	30	Attributes:
29		chunk (list): The rows returned by pagination
30		next_batch (Any\|None): Token to fetch next set of results with, if
	31	chunk: The rows returned by pagination
	32	next_batch: Token to fetch next set of results with, if
31	33	None then there are no more results.
32		prev_batch (Any\|None): Token to fetch previous set of results with, if
	34	prev_batch: Token to fetch previous set of results with, if
33	35	None then there are no previous results.
34	36	"""
35	37
36		chunk = attr.ib()
37		next_batch = attr.ib(default=None)
38		prev_batch = attr.ib(default=None)
	38	chunk = attr.ib(type=List[JsonDict])
	39	next_batch = attr.ib(type=Optional[Any], default=None)
	40	prev_batch = attr.ib(type=Optional[Any], default=None)
39	41
40		def to_dict(self):
	42	def to_dict(self) -> Dict[str, Any]:
41	43	d = {"chunk": self.chunk}
42	44
43	45	if self.next_batch:

58	60	boundaries of the chunk as pagination tokens.
59	61
60	62	Attributes:
61		topological (int): The topological ordering of the boundary event
62		stream (int): The stream ordering of the boundary event.
	63	topological: The topological ordering of the boundary event
	64	stream: The stream ordering of the boundary event.
63	65	"""
64	66
65		topological = attr.ib()
66		stream = attr.ib()
	67	topological = attr.ib(type=int)
	68	stream = attr.ib(type=int)
67	69
68	70	@staticmethod
69		def from_string(string):
	71	def from_string(string: str) -> "RelationPaginationToken":
70	72	try:
71	73	t, s = string.split("-")
72	74	return RelationPaginationToken(int(t), int(s))
73	75	except ValueError:
74	76	raise SynapseError(400, "Invalid token")
75	77
76		def to_string(self):
	78	def to_string(self) -> str:
77	79	return "%d-%d" % (self.topological, self.stream)
78	80
79		def as_tuple(self):
	81	def as_tuple(self) -> Tuple[Any, ...]:
80	82	return attr.astuple(self)
81	83
82	84

88	90	aggregation groups, we can just use them as our pagination token.
89	91
90	92	Attributes:
91		count (int): The count of relations in the boundar group.
92		stream (int): The MAX stream ordering in the boundary group.
	93	count: The count of relations in the boundary group.
	94	stream: The MAX stream ordering in the boundary group.
93	95	"""
94	96
95		count = attr.ib()
96		stream = attr.ib()
	97	count = attr.ib(type=int)
	98	stream = attr.ib(type=int)
97	99
98	100	@staticmethod
99		def from_string(string):
	101	def from_string(string: str) -> "AggregationPaginationToken":
100	102	try:
101	103	c, s = string.split("-")
102	104	return AggregationPaginationToken(int(c), int(s))
103	105	except ValueError:
104	106	raise SynapseError(400, "Invalid token")
105	107
106		def to_string(self):
	108	def to_string(self) -> str:
107	109	return "%d-%d" % (self.count, self.stream)
108	110
109		def as_tuple(self):
	111	def as_tuple(self) -> Tuple[Any, ...]:
110	112	return attr.astuple(self)

+25

-10

synapse/storage/state.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	import logging
16		from typing import Awaitable, Dict, Iterable, List, Optional, Set, Tuple, TypeVar
	15	from typing import (
	16	TYPE_CHECKING,
	17	Awaitable,
	18	Dict,
	19	Iterable,
	20	List,
	21	Optional,
	22	Set,
	23	Tuple,
	24	TypeVar,
	25	)
17	26
18	27	import attr
19	28
20	29	from synapse.api.constants import EventTypes
21	30	from synapse.events import EventBase
22	31	from synapse.types import MutableStateMap, StateMap
	32
	33	if TYPE_CHECKING:
	34	from synapse.app.homeserver import HomeServer
	35	from synapse.storage.databases import Databases
23	36
24	37	logger = logging.getLogger(__name__)
25	38

329	342	"""High level interface to fetching state for event.
330	343	"""
331	344
332		def __init__(self, hs, stores):
	345	def __init__(self, hs: "HomeServer", stores: "Databases"):
333	346	self.stores = stores
334	347
335		async def get_state_group_delta(self, state_group: int):
	348	async def get_state_group_delta(
	349	self, state_group: int
	350	) -> Tuple[Optional[int], Optional[StateMap[str]]]:
336	351	"""Given a state group try to return a previous group and a delta between
337	352	the old and the new.
338	353

340	355	state_group: The state group used to retrieve state deltas.
341	356
342	357	Returns:
343		Tuple[Optional[int], Optional[StateMap[str]]]:
344		(prev_group, delta_ids)
	358	A tuple of the previous group and a state map of the event IDs which
	359	make up the delta between the old and new state groups.
345	360	"""
346	361
347	362	return await self.stores.state.get_state_group_delta(state_group)

435	450
436	451	async def get_state_for_events(
437	452	self, event_ids: List[str], state_filter: StateFilter = StateFilter.all()
438		):
	453	) -> Dict[str, StateMap[EventBase]]:
439	454	"""Given a list of event_ids and type tuples, return a list of state
440	455	dicts for each event.
441	456

471	486
472	487	async def get_state_ids_for_events(
473	488	self, event_ids: List[str], state_filter: StateFilter = StateFilter.all()
474		):
	489	) -> Dict[str, StateMap[str]]:
475	490	"""
476	491	Get the state dicts corresponding to a list of events, containing the event_ids
477	492	of the state events (as opposed to the events themselves)

499	514
500	515	async def get_state_for_event(
501	516	self, event_id: str, state_filter: StateFilter = StateFilter.all()
502		):
	517	) -> StateMap[EventBase]:
503	518	"""
504	519	Get the state dict corresponding to a particular event
505	520

515	530
516	531	async def get_state_ids_for_event(
517	532	self, event_id: str, state_filter: StateFilter = StateFilter.all()
518		):
	533	) -> StateMap[str]:
519	534	"""
520	535	Get the state dict corresponding to a particular event
521	536

+2

-2

synapse/storage/util/id_generators.py less more

152	152
153	153	return _AsyncCtxManagerWrapper(manager())
154	154
155		def get_current_token(self):
	155	def get_current_token(self) -> int:
156	156	"""Returns the maximum stream id such that all stream ids less than or
157	157	equal to it have been successfully persisted.
158	158
159	159	Returns:
160		int
	160	The maximum stream id.
161	161	"""
162	162	with self._lock:
163	163	if self._unfinished_ids:

+5

-3

synapse/types.py less more

348	348	)
349	349
350	350
351		def map_username_to_mxid_localpart(username, case_sensitive=False):
	351	def map_username_to_mxid_localpart(
	352	username: Union[str, bytes], case_sensitive: bool = False
	353	) -> str:
352	354	"""Map a username onto a string suitable for a MXID
353	355
354	356	This follows the algorithm laid out at
355	357	https://matrix.org/docs/spec/appendices.html#mapping-from-other-character-sets.
356	358
357	359	Args:
358		username (unicode\|bytes): username to be mapped
359		case_sensitive (bool): true if TEST and test should be mapped
	360	username: username to be mapped
	361	case_sensitive: true if TEST and test should be mapped
360	362	onto different mxids
361	363
362	364	Returns:

+5

-3

synapse/util/async_helpers.py less more

14	14	# limitations under the License.
15	15
16	16	import collections
	17	import inspect
17	18	import logging
18	19	from contextlib import contextmanager
19	20	from typing import (
20	21	Any,
	22	Awaitable,
21	23	Callable,
22	24	Dict,
23	25	Hashable,

541	543	raise StopIteration(self.value)
542	544
543	545
544		def maybe_awaitable(value):
	546	def maybe_awaitable(value: Union[Awaitable[R], R]) -> Awaitable[R]:
545	547	"""Convert a value to an awaitable if not already an awaitable.
546	548	"""
547
548		if hasattr(value, "__await__"):
	549	if inspect.isawaitable(value):
	550	assert isinstance(value, Awaitable)
549	551	return value
550	552
551	553	return DoneAwaitable(value)

+2

-5

synapse/util/distributor.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		import inspect
15	14	import logging
16	15
17	16	from twisted.internet import defer
18	17
19	18	from synapse.logging.context import make_deferred_yieldable, run_in_background
20	19	from synapse.metrics.background_process_metrics import run_as_background_process
	20	from synapse.util.async_helpers import maybe_awaitable
21	21
22	22	logger = logging.getLogger(__name__)
23	23

104	104
105	105	async def do(observer):
106	106	try:
107		result = observer(args, *kwargs)
108		if inspect.isawaitable(result):
109		result = await result
110		return result
	107	return await maybe_awaitable(observer(args, *kwargs))
111	108	except Exception as e:
112	109	logger.warning(
113	110	"%s signal observer %s failed: %r", self.name, observer, e,

+58

-6

synapse/util/module_loader.py less more

14	14
15	15	import importlib
16	16	import importlib.util
	17	import itertools
	18	from typing import Any, Iterable, Tuple, Type
	19
	20	import jsonschema
17	21
18	22	from synapse.config._base import ConfigError
	23	from synapse.config._util import json_error_to_config_error
19	24
20	25
21		def load_module(provider):
	26	def load_module(provider: dict, config_path: Iterable[str]) -> Tuple[Type, Any]:
22	27	""" Loads a synapse module with its config
23		Take a dict with keys 'module' (the module name) and 'config'
24		(the config dict).
	28
	29	Args:
	30	provider: a dict with keys 'module' (the module name) and 'config'
	31	(the config dict).
	32	config_path: the path within the config file. This will be used as a basis
	33	for any error message.
25	34
26	35	Returns
27	36	Tuple of (provider class, parsed config object)
28	37	"""
	38
	39	modulename = provider.get("module")
	40	if not isinstance(modulename, str):
	41	raise ConfigError(
	42	"expected a string", path=itertools.chain(config_path, ("module",))
	43	)
	44
29	45	# We need to import the module, and then pick the class out of
30	46	# that, so we split based on the last dot.
31		module, clz = provider["module"].rsplit(".", 1)
	47	module, clz = modulename.rsplit(".", 1)
32	48	module = importlib.import_module(module)
33	49	provider_class = getattr(module, clz)
34	50
	51	module_config = provider.get("config")
35	52	try:
36		provider_config = provider_class.parse_config(provider.get("config"))
	53	provider_config = provider_class.parse_config(module_config)
	54	except jsonschema.ValidationError as e:
	55	raise json_error_to_config_error(e, itertools.chain(config_path, ("config",)))
	56	except ConfigError as e:
	57	raise _wrap_config_error(
	58	"Failed to parse config for module %r" % (modulename,),
	59	prefix=itertools.chain(config_path, ("config",)),
	60	e=e,
	61	)
37	62	except Exception as e:
38		raise ConfigError("Failed to parse config for %r: %s" % (provider["module"], e))
	63	raise ConfigError(
	64	"Failed to parse config for module %r" % (modulename,),
	65	path=itertools.chain(config_path, ("config",)),
	66	) from e
39	67
40	68	return provider_class, provider_config
41	69

55	83	mod = importlib.util.module_from_spec(spec)
56	84	spec.loader.exec_module(mod) # type: ignore
57	85	return mod
	86
	87
	88	def _wrap_config_error(
	89	msg: str, prefix: Iterable[str], e: ConfigError
	90	) -> "ConfigError":
	91	"""Wrap a relative ConfigError with a new path
	92
	93	This is useful when we have a ConfigError with a relative path due to a problem
	94	parsing part of the config, and we now need to set it in context.
	95	"""
	96	path = prefix
	97	if e.path:
	98	path = itertools.chain(prefix, e.path)
	99
	100	e1 = ConfigError(msg, path)
	101
	102	# ideally we would set the 'cause' of the new exception to the original exception;
	103	# however now that we have merged the path into our own, the stringification of
	104	# e will be incorrect, so instead we create a new exception with just the "msg"
	105	# part.
	106
	107	e1.__cause__ = Exception(e.msg)
	108	e1.__cause__.__cause__ = e.__cause__
	109	return e1

+29

-15

synapse/visibility.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14
15	14	import logging
16	15	import operator
17	16
18		from synapse.api.constants import AccountDataTypes, EventTypes, Membership
	17	from synapse.api.constants import (
	18	AccountDataTypes,
	19	EventTypes,
	20	HistoryVisibility,
	21	Membership,
	22	)
19	23	from synapse.events.utils import prune_event
20	24	from synapse.storage import Storage
21	25	from synapse.storage.state import StateFilter

24	28	logger = logging.getLogger(__name__)
25	29
26	30
27		VISIBILITY_PRIORITY = ("world_readable", "shared", "invited", "joined")
	31	VISIBILITY_PRIORITY = (
	32	HistoryVisibility.WORLD_READABLE,
	33	HistoryVisibility.SHARED,
	34	HistoryVisibility.INVITED,
	35	HistoryVisibility.JOINED,
	36	)
28	37
29	38
30	39	MEMBERSHIP_PRIORITY = (

115	124	# see events in the room at that point in the DAG, and that shouldn't be decided
116	125	# on those checks.
117	126	if filter_send_to_client:
118		if event.type == "org.matrix.dummy_event":
	127	if event.type == EventTypes.Dummy:
119	128	return None
120	129
121	130	if not event.is_state() and event.sender in ignore_list:

149	158	# get the room_visibility at the time of the event.
150	159	visibility_event = state.get((EventTypes.RoomHistoryVisibility, ""), None)
151	160	if visibility_event:
152		visibility = visibility_event.content.get("history_visibility", "shared")
	161	visibility = visibility_event.content.get(
	162	"history_visibility", HistoryVisibility.SHARED
	163	)
153	164	else:
154		visibility = "shared"
	165	visibility = HistoryVisibility.SHARED
155	166
156	167	if visibility not in VISIBILITY_PRIORITY:
157		visibility = "shared"
	168	visibility = HistoryVisibility.SHARED
158	169
159	170	# Always allow history visibility events on boundaries. This is done
160	171	# by setting the effective visibility to the least restrictive

164	175	prev_visibility = prev_content.get("history_visibility", None)
165	176
166	177	if prev_visibility not in VISIBILITY_PRIORITY:
167		prev_visibility = "shared"
	178	prev_visibility = HistoryVisibility.SHARED
168	179
169	180	new_priority = VISIBILITY_PRIORITY.index(visibility)
170	181	old_priority = VISIBILITY_PRIORITY.index(prev_visibility)

209	220
210	221	# otherwise, it depends on the room visibility.
211	222
212		if visibility == "joined":
	223	if visibility == HistoryVisibility.JOINED:
213	224	# we weren't a member at the time of the event, so we can't
214	225	# see this event.
215	226	return None
216	227
217		elif visibility == "invited":
	228	elif visibility == HistoryVisibility.INVITED:
218	229	# user can also see the event if they were invited at the time
219	230	# of the event.
220	231	return event if membership == Membership.INVITE else None
221	232
222		elif visibility == "shared" and is_peeking:
	233	elif visibility == HistoryVisibility.SHARED and is_peeking:
223	234	# if the visibility is shared, users cannot see the event unless
224	235	# they have subequently joined the room (or were members at the
225	236	# time, of course)

283	294	def check_event_is_visible(event, state):
284	295	history = state.get((EventTypes.RoomHistoryVisibility, ""), None)
285	296	if history:
286		visibility = history.content.get("history_visibility", "shared")
287		if visibility in ["invited", "joined"]:
	297	visibility = history.content.get(
	298	"history_visibility", HistoryVisibility.SHARED
	299	)
	300	if visibility in [HistoryVisibility.INVITED, HistoryVisibility.JOINED]:
288	301	# We now loop through all state events looking for
289	302	# membership states for the requesting server to determine
290	303	# if the server is either in the room or has been invited

304	317	if memtype == Membership.JOIN:
305	318	return True
306	319	elif memtype == Membership.INVITE:
307		if visibility == "invited":
	320	if visibility == HistoryVisibility.INVITED:
308	321	return True
309	322	else:
310	323	# server has no users in the room: redact

335	348	else:
336	349	event_map = await storage.main.get_events(visibility_ids)
337	350	all_open = all(
338		e.content.get("history_visibility") in (None, "shared", "world_readable")
	351	e.content.get("history_visibility")
	352	in (None, HistoryVisibility.SHARED, HistoryVisibility.WORLD_READABLE)
339	353	for e in event_map.values()
340	354	)
341	355

+2

-14

tests/api/test_filtering.py less more

15	15	# See the License for the specific language governing permissions and
16	16	# limitations under the License.
17	17
18		from mock import Mock
19
20	18	import jsonschema
21	19
22	20	from twisted.internet import defer

27	25	from synapse.events import make_event_from_dict
28	26
29	27	from tests import unittest
30		from tests.utils import DeferredMockCallable, MockHttpResource, setup_test_homeserver
	28	from tests.utils import setup_test_homeserver
31	29
32	30	user_localpart = "test_user"
33	31

41	39
42	40
43	41	class FilteringTestCase(unittest.TestCase):
44		@defer.inlineCallbacks
45	42	def setUp(self):
46		self.mock_federation_resource = MockHttpResource()
47
48		self.mock_http_client = Mock(spec=[])
49		self.mock_http_client.put_json = DeferredMockCallable()
50
51		hs = yield setup_test_homeserver(
52		self.addCleanup, http_client=self.mock_http_client, keyring=Mock(),
53		)
54
	43	hs = setup_test_homeserver(self.addCleanup)
55	44	self.filtering = hs.get_filtering()
56
57	45	self.datastore = hs.get_datastore()
58	46
59	47	def test_errors_on_invalid_filters(self):

+3

-3

tests/app/test_frontend_proxy.py less more

22	22	def make_homeserver(self, reactor, clock):
23	23
24	24	hs = self.setup_test_homeserver(
25		http_client=None, homeserver_to_use=GenericWorkerServer
	25	federation_http_client=None, homeserver_to_use=GenericWorkerServer
26	26	)
27	27
28	28	return hs

56	56	self.assertEqual(len(self.reactor.tcpServers), 1)
57	57	site = self.reactor.tcpServers[0][1]
58	58
59		_, channel = make_request(self.reactor, site, "PUT", "presence/a/status")
	59	channel = make_request(self.reactor, site, "PUT", "presence/a/status")
60	60
61	61	# 400 + unrecognised, because nothing is registered
62	62	self.assertEqual(channel.code, 400)

76	76	self.assertEqual(len(self.reactor.tcpServers), 1)
77	77	site = self.reactor.tcpServers[0][1]
78	78
79		_, channel = make_request(self.reactor, site, "PUT", "presence/a/status")
	79	channel = make_request(self.reactor, site, "PUT", "presence/a/status")
80	80
81	81	# 401, because the stub servlet still checks authentication
82	82	self.assertEqual(channel.code, 401)

+4

-4

tests/app/test_openid_listener.py less more

26	26	class FederationReaderOpenIDListenerTests(HomeserverTestCase):
27	27	def make_homeserver(self, reactor, clock):
28	28	hs = self.setup_test_homeserver(
29		http_client=None, homeserver_to_use=GenericWorkerServer
	29	federation_http_client=None, homeserver_to_use=GenericWorkerServer
30	30	)
31	31	return hs
32	32

72	72	return
73	73	raise
74	74
75		_, channel = make_request(
	75	channel = make_request(
76	76	self.reactor, site, "GET", "/_matrix/federation/v1/openid/userinfo"
77	77	)
78	78

83	83	class SynapseHomeserverOpenIDListenerTests(HomeserverTestCase):
84	84	def make_homeserver(self, reactor, clock):
85	85	hs = self.setup_test_homeserver(
86		http_client=None, homeserver_to_use=SynapseHomeServer
	86	federation_http_client=None, homeserver_to_use=SynapseHomeServer
87	87	)
88	88	return hs
89	89

120	120	return
121	121	raise
122	122
123		_, channel = make_request(
	123	channel = make_request(
124	124	self.reactor, site, "GET", "/_matrix/federation/v1/openid/userinfo"
125	125	)
126	126

+9

-7

tests/crypto/test_keyring.py less more

74	74	return val
75	75
76	76	def test_verify_json_objects_for_server_awaits_previous_requests(self):
77		mock_fetcher = keyring.KeyFetcher()
	77	mock_fetcher = Mock()
78	78	mock_fetcher.get_keys = Mock()
79	79	kr = keyring.Keyring(self.hs, key_fetchers=(mock_fetcher,))
80	80

194	194	"""Tests that we correctly handle key requests for keys we've stored
195	195	with a null `ts_valid_until_ms`
196	196	"""
197		mock_fetcher = keyring.KeyFetcher()
	197	mock_fetcher = Mock()
198	198	mock_fetcher.get_keys = Mock(return_value=make_awaitable({}))
199	199
200	200	kr = keyring.Keyring(

248	248	}
249	249	}
250	250
251		mock_fetcher = keyring.KeyFetcher()
	251	mock_fetcher = Mock()
252	252	mock_fetcher.get_keys = Mock(side_effect=get_keys)
253	253	kr = keyring.Keyring(self.hs, key_fetchers=(mock_fetcher,))
254	254

287	287	}
288	288	}
289	289
290		mock_fetcher1 = keyring.KeyFetcher()
	290	mock_fetcher1 = Mock()
291	291	mock_fetcher1.get_keys = Mock(side_effect=get_keys1)
292		mock_fetcher2 = keyring.KeyFetcher()
	292	mock_fetcher2 = Mock()
293	293	mock_fetcher2.get_keys = Mock(side_effect=get_keys2)
294	294	kr = keyring.Keyring(self.hs, key_fetchers=(mock_fetcher1, mock_fetcher2))
295	295

314	314	class ServerKeyFetcherTestCase(unittest.HomeserverTestCase):
315	315	def make_homeserver(self, reactor, clock):
316	316	self.http_client = Mock()
317		hs = self.setup_test_homeserver(http_client=self.http_client)
	317	hs = self.setup_test_homeserver(federation_http_client=self.http_client)
318	318	return hs
319	319
320	320	def test_get_keys_from_server(self):

394	394	}
395	395	]
396	396
397		return self.setup_test_homeserver(http_client=self.http_client, config=config)
	397	return self.setup_test_homeserver(
	398	federation_http_client=self.http_client, config=config
	399	)
398	400
399	401	def build_perspectives_response(
400	402	self, server_name: str, signing_key: SigningKey, valid_until_ts: int,

+2

-2

tests/federation/test_complexity.py less more

47	47	)
48	48
49	49	# Get the room complexity
50		request, channel = self.make_request(
	50	channel = self.make_request(
51	51	"GET", "/_matrix/federation/unstable/rooms/%s/complexity" % (room_1,)
52	52	)
53	53	self.assertEquals(200, channel.code)

59	59	store.get_current_state_event_counts = lambda x: make_awaitable(500 * 1.23)
60	60
61	61	# Get the room complexity again -- make sure it's our artificial value
62		request, channel = self.make_request(
	62	channel = self.make_request(
63	63	"GET", "/_matrix/federation/unstable/rooms/%s/complexity" % (room_1,)
64	64	)
65	65	self.assertEquals(200, channel.code)

+3

-3

tests/federation/test_federation_server.py less more

45	45
46	46	"/get_missing_events/(?P<room_id>[^/]*)/?"
47	47
48		request, channel = self.make_request(
	48	channel = self.make_request(
49	49	"POST",
50	50	"/_matrix/federation/v1/get_missing_events/%s" % (room_1,),
51	51	query_content,

94	94	room_1 = self.helper.create_room_as(u1, tok=u1_token)
95	95	self.inject_room_member(room_1, "@user:other.example.com", "join")
96	96
97		request, channel = self.make_request(
	97	channel = self.make_request(
98	98	"GET", "/_matrix/federation/v1/state/%s" % (room_1,)
99	99	)
100	100	self.assertEquals(200, channel.code, channel.result)

126	126
127	127	room_1 = self.helper.create_room_as(u1, tok=u1_token)
128	128
129		request, channel = self.make_request(
	129	channel = self.make_request(
130	130	"GET", "/_matrix/federation/v1/state/%s" % (room_1,)
131	131	)
132	132	self.assertEquals(403, channel.code, channel.result)

+0

-0

tests/federation/transport/__init__.py less more

(New empty file)

+16

-23

tests/federation/transport/test_server.py less more

0	0	# -- coding: utf-8 --
1		# Copyright 2019 The Matrix.org Foundation C.I.C.
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
2	2	#
3	3	# Licensed under the Apache License, Version 2.0 (the "License");
4	4	# you may not use this file except in compliance with the License.

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14
15
16		from twisted.internet import defer
17
18		from synapse.config.ratelimiting import FederationRateLimitConfig
19		from synapse.federation.transport import server
20		from synapse.util.ratelimitutils import FederationRateLimiter
21
22	15	from tests import unittest
23	16	from tests.unittest import override_config
24	17
25	18
26		class RoomDirectoryFederationTests(unittest.HomeserverTestCase):
27		def prepare(self, reactor, clock, homeserver):
28		class Authenticator:
29		def authenticate_request(self, request, content):
30		return defer.succeed("otherserver.nottld")
31
32		ratelimiter = FederationRateLimiter(clock, FederationRateLimitConfig())
33		server.register_servlets(
34		homeserver, self.resource, Authenticator(), ratelimiter
35		)
36
	19	class RoomDirectoryFederationTests(unittest.FederatingHomeserverTestCase):
37	20	@override_config({"allow_public_rooms_over_federation": False})
38	21	def test_blocked_public_room_list_over_federation(self):
39		request, channel = self.make_request(
40		"GET", "/_matrix/federation/v1/publicRooms"
	22	"""Test that unauthenticated requests to the public rooms directory 403 when
	23	allow_public_rooms_over_federation is False.
	24	"""
	25	channel = self.make_request(
	26	"GET",
	27	"/_matrix/federation/v1/publicRooms",
	28	federation_auth_origin=b"example.com",
41	29	)
42	30	self.assertEquals(403, channel.code)
43	31
44	32	@override_config({"allow_public_rooms_over_federation": True})
45	33	def test_open_public_room_list_over_federation(self):
46		request, channel = self.make_request(
47		"GET", "/_matrix/federation/v1/publicRooms"
	34	"""Test that unauthenticated requests to the public rooms directory 200 when
	35	allow_public_rooms_over_federation is True.
	36	"""
	37	channel = self.make_request(
	38	"GET",
	39	"/_matrix/federation/v1/publicRooms",
	40	federation_auth_origin=b"example.com",
48	41	)
49	42	self.assertEquals(200, channel.code)

+121

-0

tests/handlers/test_cas.py less more

	0	# Copyright 2020 The Matrix.org Foundation C.I.C.
	1	#
	2	# Licensed under the Apache License, Version 2.0 (the "License");
	3	# you may not use this file except in compliance with the License.
	4	# You may obtain a copy of the License at
	5	#
	6	# http://www.apache.org/licenses/LICENSE-2.0
	7	#
	8	# Unless required by applicable law or agreed to in writing, software
	9	# distributed under the License is distributed on an "AS IS" BASIS,
	10	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	11	# See the License for the specific language governing permissions and
	12	# limitations under the License.
	13	from mock import Mock
	14
	15	from synapse.handlers.cas_handler import CasResponse
	16
	17	from tests.test_utils import simple_async_mock
	18	from tests.unittest import HomeserverTestCase
	19
	20	# These are a few constants that are used as config parameters in the tests.
	21	BASE_URL = "https://synapse/"
	22	SERVER_URL = "https://issuer/"
	23
	24
	25	class CasHandlerTestCase(HomeserverTestCase):
	26	def default_config(self):
	27	config = super().default_config()
	28	config["public_baseurl"] = BASE_URL
	29	cas_config = {
	30	"enabled": True,
	31	"server_url": SERVER_URL,
	32	"service_url": BASE_URL,
	33	}
	34	config["cas_config"] = cas_config
	35
	36	return config
	37
	38	def make_homeserver(self, reactor, clock):
	39	hs = self.setup_test_homeserver()
	40
	41	self.handler = hs.get_cas_handler()
	42
	43	# Reduce the number of attempts when generating MXIDs.
	44	sso_handler = hs.get_sso_handler()
	45	sso_handler._MAP_USERNAME_RETRIES = 3
	46
	47	return hs
	48
	49	def test_map_cas_user_to_user(self):
	50	"""Ensure that mapping the CAS user returned from a provider to an MXID works properly."""
	51
	52	# stub out the auth handler
	53	auth_handler = self.hs.get_auth_handler()
	54	auth_handler.complete_sso_login = simple_async_mock()
	55
	56	cas_response = CasResponse("test_user", {})
	57	request = _mock_request()
	58	self.get_success(
	59	self.handler._handle_cas_response(request, cas_response, "redirect_uri", "")
	60	)
	61
	62	# check that the auth handler got called as expected
	63	auth_handler.complete_sso_login.assert_called_once_with(
	64	"@test_user:test", request, "redirect_uri", None
	65	)
	66
	67	def test_map_cas_user_to_existing_user(self):
	68	"""Existing users can log in with CAS account."""
	69	store = self.hs.get_datastore()
	70	self.get_success(
	71	store.register_user(user_id="@test_user:test", password_hash=None)
	72	)
	73
	74	# stub out the auth handler
	75	auth_handler = self.hs.get_auth_handler()
	76	auth_handler.complete_sso_login = simple_async_mock()
	77
	78	# Map a user via SSO.
	79	cas_response = CasResponse("test_user", {})
	80	request = _mock_request()
	81	self.get_success(
	82	self.handler._handle_cas_response(request, cas_response, "redirect_uri", "")
	83	)
	84
	85	# check that the auth handler got called as expected
	86	auth_handler.complete_sso_login.assert_called_once_with(
	87	"@test_user:test", request, "redirect_uri", None
	88	)
	89
	90	# Subsequent calls should map to the same mxid.
	91	auth_handler.complete_sso_login.reset_mock()
	92	self.get_success(
	93	self.handler._handle_cas_response(request, cas_response, "redirect_uri", "")
	94	)
	95	auth_handler.complete_sso_login.assert_called_once_with(
	96	"@test_user:test", request, "redirect_uri", None
	97	)
	98
	99	def test_map_cas_user_to_invalid_localpart(self):
	100	"""CAS automaps invalid characters to base-64 encoding."""
	101
	102	# stub out the auth handler
	103	auth_handler = self.hs.get_auth_handler()
	104	auth_handler.complete_sso_login = simple_async_mock()
	105
	106	cas_response = CasResponse("föö", {})
	107	request = _mock_request()
	108	self.get_success(
	109	self.handler._handle_cas_response(request, cas_response, "redirect_uri", "")
	110	)
	111
	112	# check that the auth handler got called as expected
	113	auth_handler.complete_sso_login.assert_called_once_with(
	114	"@f=c3=b6=c3=b6:test", request, "redirect_uri", None
	115	)
	116
	117
	118	def _mock_request():
	119	"""Returns a mock which will stand in as a SynapseRequest"""
	120	return Mock(spec=["getClientIP", "get_user_agent"])

+2

-2

tests/handlers/test_device.py less more

26	26
27	27	class DeviceTestCase(unittest.HomeserverTestCase):
28	28	def make_homeserver(self, reactor, clock):
29		hs = self.setup_test_homeserver("server", http_client=None)
	29	hs = self.setup_test_homeserver("server", federation_http_client=None)
30	30	self.handler = hs.get_device_handler()
31	31	self.store = hs.get_datastore()
32	32	return hs

228	228
229	229	class DehydrationTestCase(unittest.HomeserverTestCase):
230	230	def make_homeserver(self, reactor, clock):
231		hs = self.setup_test_homeserver("server", http_client=None)
	231	hs = self.setup_test_homeserver("server", federation_http_client=None)
232	232	self.handler = hs.get_device_handler()
233	233	self.registration = hs.get_registration_handler()
234	234	self.auth = hs.get_auth()

+6

-8

tests/handlers/test_directory.py less more

41	41	self.mock_registry.register_query_handler = register_query_handler
42	42
43	43	hs = self.setup_test_homeserver(
44		http_client=None,
45		resource_for_federation=Mock(),
46	44	federation_client=self.mock_federation,
47	45	federation_registry=self.mock_registry,
48	46	)

406	404	def test_denied(self):
407	405	room_id = self.helper.create_room_as(self.user_id)
408	406
409		request, channel = self.make_request(
	407	channel = self.make_request(
410	408	"PUT",
411	409	b"directory/room/%23test%3Atest",
412	410	('{"room_id":"%s"}' % (room_id,)).encode("ascii"),

416	414	def test_allowed(self):
417	415	room_id = self.helper.create_room_as(self.user_id)
418	416
419		request, channel = self.make_request(
	417	channel = self.make_request(
420	418	"PUT",
421	419	b"directory/room/%23unofficial_test%3Atest",
422	420	('{"room_id":"%s"}' % (room_id,)).encode("ascii"),

432	430	def prepare(self, reactor, clock, hs):
433	431	room_id = self.helper.create_room_as(self.user_id)
434	432
435		request, channel = self.make_request(
	433	channel = self.make_request(
436	434	"PUT", b"directory/list/room/%s" % (room_id.encode("ascii"),), b"{}"
437	435	)
438	436	self.assertEquals(200, channel.code, channel.result)

447	445	self.directory_handler.enable_room_list_search = True
448	446
449	447	# Room list is enabled so we should get some results
450		request, channel = self.make_request("GET", b"publicRooms")
	448	channel = self.make_request("GET", b"publicRooms")
451	449	self.assertEquals(200, channel.code, channel.result)
452	450	self.assertTrue(len(channel.json_body["chunk"]) > 0)
453	451

455	453	self.directory_handler.enable_room_list_search = False
456	454
457	455	# Room list disabled so we should get no results
458		request, channel = self.make_request("GET", b"publicRooms")
	456	channel = self.make_request("GET", b"publicRooms")
459	457	self.assertEquals(200, channel.code, channel.result)
460	458	self.assertTrue(len(channel.json_body["chunk"]) == 0)
461	459
462	460	# Room list disabled so we shouldn't be allowed to publish rooms
463	461	room_id = self.helper.create_room_as(self.user_id)
464		request, channel = self.make_request(
	462	channel = self.make_request(
465	463	"PUT", b"directory/list/room/%s" % (room_id.encode("ascii"),), b"{}"
466	464	)
467	465	self.assertEquals(403, channel.code, channel.result)

+4

-4

tests/handlers/test_federation.py less more

36	36	]
37	37
38	38	def make_homeserver(self, reactor, clock):
39		hs = self.setup_test_homeserver(http_client=None)
	39	hs = self.setup_test_homeserver(federation_http_client=None)
40	40	self.handler = hs.get_federation_handler()
41	41	self.store = hs.get_datastore()
42	42	return hs

125	125	room_version,
126	126	)
127	127
128		with LoggingContext(request="send_rejected"):
	128	with LoggingContext("send_rejected"):
129	129	d = run_in_background(self.handler.on_receive_pdu, OTHER_SERVER, ev)
130	130	self.get_success(d)
131	131

177	177	room_version,
178	178	)
179	179
180		with LoggingContext(request="send_rejected"):
	180	with LoggingContext("send_rejected"):
181	181	d = run_in_background(self.handler.on_receive_pdu, OTHER_SERVER, ev)
182	182	self.get_success(d)
183	183

197	197	# the auth code requires that a signature exists, but doesn't check that
198	198	# signature... go figure.
199	199	join_event.signatures[other_server] = {"x": "y"}
200		with LoggingContext(request="send_join"):
	200	with LoggingContext("send_join"):
201	201	d = run_in_background(
202	202	self.handler.on_send_join_request, other_server, join_event
203	203	)

+1

-1

tests/handlers/test_message.py less more

205	205
206	206	# Redaction of event should fail.
207	207	path = "/_matrix/client/r0/rooms/%s/redact/%s" % (self.room_id, event_id)
208		request, channel = self.make_request(
	208	channel = self.make_request(
209	209	"POST", path, content={}, access_token=self.access_token
210	210	)
211	211	self.assertEqual(int(channel.result["code"]), 403)

+327

-184

tests/handlers/test_oidc.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import json
15		from urllib.parse import parse_qs, urlparse
16
17		from mock import Mock, patch
18
19		import attr
	15	import re
	16	from typing import Dict
	17	from urllib.parse import parse_qs, urlencode, urlparse
	18
	19	from mock import ANY, Mock, patch
	20
20	21	import pymacaroons
21	22
22		from twisted.python.failure import Failure
23		from twisted.web._newclient import ResponseDone
24
25		from synapse.handlers.oidc_handler import OidcError, OidcMappingProvider
	23	from twisted.web.resource import Resource
	24
	25	from synapse.api.errors import RedirectException
	26	from synapse.handlers.oidc_handler import OidcError
26	27	from synapse.handlers.sso import MappingException
	28	from synapse.rest.client.v1 import login
	29	from synapse.rest.synapse.client.pick_username import pick_username_resource
	30	from synapse.server import HomeServer
27	31	from synapse.types import UserID
28	32
	33	from tests.test_utils import FakeResponse, simple_async_mock
29	34	from tests.unittest import HomeserverTestCase, override_config
30
31
32		@attr.s
33		class FakeResponse:
34		code = attr.ib()
35		body = attr.ib()
36		phrase = attr.ib()
37
38		def deliverBody(self, protocol):
39		protocol.dataReceived(self.body)
40		protocol.connectionLost(Failure(ResponseDone()))
41
42	35
43	36	# These are a few constants that are used as config parameters in the tests.
44	37	ISSUER = "https://issuer/"

69	62	COOKIE_PATH = "/_synapse/oidc"
70	63
71	64
72		class TestMappingProvider(OidcMappingProvider):
	65	class TestMappingProvider:
73	66	@staticmethod
74	67	def parse_config(config):
75	68	return
	69
	70	def __init__(self, config):
	71	pass
76	72
77	73	def get_remote_user_id(self, userinfo):
78	74	return userinfo["sub"]

94	90	"localpart": userinfo["username"] + (str(failures) if failures else ""),
95	91	"display_name": None,
96	92	}
97
98
99		def simple_async_mock(return_value=None, raises=None):
100		# AsyncMock is not available in python3.5, this mimics part of its behaviour
101		async def cb(args, *kwargs):
102		if raises:
103		raise raises
104		return return_value
105
106		return Mock(side_effect=cb)
107	93
108	94
109	95	async def get_json(url):

174	160	self.assertEqual(args[2], error_description)
175	161	# Reset the render_error mock
176	162	self.render_error.reset_mock()
	163	return args
177	164
178	165	def test_config(self):
179	166	"""Basic config correctly sets up the callback URL and client auth correctly."""

383	370	- when the userinfo fetching fails
384	371	- when the code exchange fails
385	372	"""
	373
	374	# ensure that we are correctly testing the fallback when "get_extra_attributes"
	375	# is not implemented.
	376	mapping_provider = self.handler._user_mapping_provider
	377	with self.assertRaises(AttributeError):
	378	_ = mapping_provider.get_extra_attributes
	379
386	380	token = {
387	381	"type": "bearer",
388	382	"id_token": "id_token",
389	383	"access_token": "access_token",
390	384	}
	385	username = "bar"
391	386	userinfo = {
392	387	"sub": "foo",
393		"preferred_username": "bar",
394		}
395		user_id = "@foo:domain.org"
	388	"username": username,
	389	}
	390	expected_user_id = "@%s:%s" % (username, self.hs.hostname)
396	391	self.handler._exchange_code = simple_async_mock(return_value=token)
397	392	self.handler._parse_id_token = simple_async_mock(return_value=userinfo)
398	393	self.handler._fetch_userinfo = simple_async_mock(return_value=userinfo)
399		self.handler._map_userinfo_to_user = simple_async_mock(return_value=user_id)
400		self.handler._auth_handler.complete_sso_login = simple_async_mock()
401		request = Mock(
402		spec=[
403		"args",
404		"getCookie",
405		"addCookie",
406		"requestHeaders",
407		"getClientIP",
408		"get_user_agent",
409		]
410		)
	394	auth_handler = self.hs.get_auth_handler()
	395	auth_handler.complete_sso_login = simple_async_mock()
411	396
412	397	code = "code"
413	398	state = "state"

415	400	client_redirect_url = "http://client/redirect"
416	401	user_agent = "Browser"
417	402	ip_address = "10.0.0.1"
418		request.getCookie.return_value = self.handler._generate_oidc_session_token(
	403	session = self.handler._generate_oidc_session_token(
419	404	state=state,
420	405	nonce=nonce,
421	406	client_redirect_url=client_redirect_url,
422	407	ui_auth_session_id=None,
423	408	)
424
425		request.args = {}
426		request.args[b"code"] = [code.encode("utf-8")]
427		request.args[b"state"] = [state.encode("utf-8")]
428
429		request.getClientIP.return_value = ip_address
430		request.get_user_agent.return_value = user_agent
431
432		self.get_success(self.handler.handle_oidc_callback(request))
433
434		self.handler._auth_handler.complete_sso_login.assert_called_once_with(
435		user_id, request, client_redirect_url, {},
	409	request = _build_callback_request(
	410	code, state, session, user_agent=user_agent, ip_address=ip_address
	411	)
	412
	413	self.get_success(self.handler.handle_oidc_callback(request))
	414
	415	auth_handler.complete_sso_login.assert_called_once_with(
	416	expected_user_id, request, client_redirect_url, None,
436	417	)
437	418	self.handler._exchange_code.assert_called_once_with(code)
438	419	self.handler._parse_id_token.assert_called_once_with(token, nonce=nonce)
439		self.handler._map_userinfo_to_user.assert_called_once_with(
440		userinfo, token, user_agent, ip_address
441		)
442	420	self.handler._fetch_userinfo.assert_not_called()
443	421	self.render_error.assert_not_called()
444	422
445	423	# Handle mapping errors
446		self.handler._map_userinfo_to_user = simple_async_mock(
447		raises=MappingException()
448		)
449		self.get_success(self.handler.handle_oidc_callback(request))
450		self.assertRenderedError("mapping_error")
451		self.handler._map_userinfo_to_user = simple_async_mock(return_value=user_id)
	424	with patch.object(
	425	self.handler,
	426	"_remote_id_from_userinfo",
	427	new=Mock(side_effect=MappingException()),
	428	):
	429	self.get_success(self.handler.handle_oidc_callback(request))
	430	self.assertRenderedError("mapping_error")
452	431
453	432	# Handle ID token errors
454	433	self.handler._parse_id_token = simple_async_mock(raises=Exception())
455	434	self.get_success(self.handler.handle_oidc_callback(request))
456	435	self.assertRenderedError("invalid_token")
457	436
458		self.handler._auth_handler.complete_sso_login.reset_mock()
	437	auth_handler.complete_sso_login.reset_mock()
459	438	self.handler._exchange_code.reset_mock()
460	439	self.handler._parse_id_token.reset_mock()
461		self.handler._map_userinfo_to_user.reset_mock()
462	440	self.handler._fetch_userinfo.reset_mock()
463	441
464	442	# With userinfo fetching
465	443	self.handler._scopes = [] # do not ask the "openid" scope
466	444	self.get_success(self.handler.handle_oidc_callback(request))
467	445
468		self.handler._auth_handler.complete_sso_login.assert_called_once_with(
469		user_id, request, client_redirect_url, {},
	446	auth_handler.complete_sso_login.assert_called_once_with(
	447	expected_user_id, request, client_redirect_url, None,
470	448	)
471	449	self.handler._exchange_code.assert_called_once_with(code)
472	450	self.handler._parse_id_token.assert_not_called()
473		self.handler._map_userinfo_to_user.assert_called_once_with(
474		userinfo, token, user_agent, ip_address
475		)
476	451	self.handler._fetch_userinfo.assert_called_once_with(token)
477	452	self.render_error.assert_not_called()
478	453

623	598	}
624	599	userinfo = {
625	600	"sub": "foo",
	601	"username": "foo",
626	602	"phone": "1234567",
627	603	}
628		user_id = "@foo:domain.org"
629	604	self.handler._exchange_code = simple_async_mock(return_value=token)
630	605	self.handler._parse_id_token = simple_async_mock(return_value=userinfo)
631		self.handler._map_userinfo_to_user = simple_async_mock(return_value=user_id)
632		self.handler._auth_handler.complete_sso_login = simple_async_mock()
633		request = Mock(
634		spec=[
635		"args",
636		"getCookie",
637		"addCookie",
638		"requestHeaders",
639		"getClientIP",
640		"get_user_agent",
641		]
642		)
	606	auth_handler = self.hs.get_auth_handler()
	607	auth_handler.complete_sso_login = simple_async_mock()
643	608
644	609	state = "state"
645	610	client_redirect_url = "http://client/redirect"
646		request.getCookie.return_value = self.handler._generate_oidc_session_token(
	611	session = self.handler._generate_oidc_session_token(
647	612	state=state,
648	613	nonce="nonce",
649	614	client_redirect_url=client_redirect_url,
650	615	ui_auth_session_id=None,
651	616	)
652
653		request.args = {}
654		request.args[b"code"] = [b"code"]
655		request.args[b"state"] = [state.encode("utf-8")]
656
657		request.getClientIP.return_value = "10.0.0.1"
658		request.get_user_agent.return_value = "Browser"
659
660		self.get_success(self.handler.handle_oidc_callback(request))
661
662		self.handler._auth_handler.complete_sso_login.assert_called_once_with(
663		user_id, request, client_redirect_url, {"phone": "1234567"},
	617	request = _build_callback_request("code", state, session)
	618
	619	self.get_success(self.handler.handle_oidc_callback(request))
	620
	621	auth_handler.complete_sso_login.assert_called_once_with(
	622	"@foo:test", request, client_redirect_url, {"phone": "1234567"},
664	623	)
665	624
666	625	def test_map_userinfo_to_user(self):
667	626	"""Ensure that mapping the userinfo returned from a provider to an MXID works properly."""
	627	auth_handler = self.hs.get_auth_handler()
	628	auth_handler.complete_sso_login = simple_async_mock()
	629
668	630	userinfo = {
669	631	"sub": "test_user",
670	632	"username": "test_user",
671	633	}
672		# The token doesn't matter with the default user mapping provider.
673		token = {}
674		mxid = self.get_success(
675		self.handler._map_userinfo_to_user(
676		userinfo, token, "user-agent", "10.10.10.10"
677		)
678		)
679		self.assertEqual(mxid, "@test_user:test")
	634	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	635	auth_handler.complete_sso_login.assert_called_once_with(
	636	"@test_user:test", ANY, ANY, None,
	637	)
	638	auth_handler.complete_sso_login.reset_mock()
680	639
681	640	# Some providers return an integer ID.
682	641	userinfo = {
683	642	"sub": 1234,
684	643	"username": "test_user_2",
685	644	}
686		mxid = self.get_success(
687		self.handler._map_userinfo_to_user(
688		userinfo, token, "user-agent", "10.10.10.10"
689		)
690		)
691		self.assertEqual(mxid, "@test_user_2:test")
	645	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	646	auth_handler.complete_sso_login.assert_called_once_with(
	647	"@test_user_2:test", ANY, ANY, None,
	648	)
	649	auth_handler.complete_sso_login.reset_mock()
692	650
693	651	# Test if the mxid is already taken
694	652	store = self.hs.get_datastore()

697	655	store.register_user(user_id=user3.to_string(), password_hash=None)
698	656	)
699	657	userinfo = {"sub": "test3", "username": "test_user_3"}
700		e = self.get_failure(
701		self.handler._map_userinfo_to_user(
702		userinfo, token, "user-agent", "10.10.10.10"
703		),
704		MappingException,
705		)
706		self.assertEqual(
707		str(e.value), "Mapping provider does not support de-duplicating Matrix IDs",
	658	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	659	auth_handler.complete_sso_login.assert_not_called()
	660	self.assertRenderedError(
	661	"mapping_error",
	662	"Mapping provider does not support de-duplicating Matrix IDs",
708	663	)
709	664
710	665	@override_config({"oidc_config": {"allow_existing_users": True}})

716	671	store.register_user(user_id=user.to_string(), password_hash=None)
717	672	)
718	673
	674	auth_handler = self.hs.get_auth_handler()
	675	auth_handler.complete_sso_login = simple_async_mock()
	676
719	677	# Map a user via SSO.
720	678	userinfo = {
721	679	"sub": "test",
722	680	"username": "test_user",
723	681	}
724		token = {}
725		mxid = self.get_success(
726		self.handler._map_userinfo_to_user(
727		userinfo, token, "user-agent", "10.10.10.10"
728		)
729		)
730		self.assertEqual(mxid, "@test_user:test")
	682	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	683	auth_handler.complete_sso_login.assert_called_once_with(
	684	user.to_string(), ANY, ANY, None,
	685	)
	686	auth_handler.complete_sso_login.reset_mock()
731	687
732	688	# Subsequent calls should map to the same mxid.
733		mxid = self.get_success(
734		self.handler._map_userinfo_to_user(
735		userinfo, token, "user-agent", "10.10.10.10"
736		)
737		)
738		self.assertEqual(mxid, "@test_user:test")
	689	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	690	auth_handler.complete_sso_login.assert_called_once_with(
	691	user.to_string(), ANY, ANY, None,
	692	)
	693	auth_handler.complete_sso_login.reset_mock()
739	694
740	695	# Note that a second SSO user can be mapped to the same Matrix ID. (This
741	696	# requires a unique sub, but something that maps to the same matrix ID,

746	701	"sub": "test1",
747	702	"username": "test_user",
748	703	}
749		token = {}
750		mxid = self.get_success(
751		self.handler._map_userinfo_to_user(
752		userinfo, token, "user-agent", "10.10.10.10"
753		)
754		)
755		self.assertEqual(mxid, "@test_user:test")
	704	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	705	auth_handler.complete_sso_login.assert_called_once_with(
	706	user.to_string(), ANY, ANY, None,
	707	)
	708	auth_handler.complete_sso_login.reset_mock()
756	709
757	710	# Register some non-exact matching cases.
758	711	user2 = UserID.from_string("@TEST_user_2:test")

769	722	"sub": "test2",
770	723	"username": "TEST_USER_2",
771	724	}
772		e = self.get_failure(
773		self.handler._map_userinfo_to_user(
774		userinfo, token, "user-agent", "10.10.10.10"
775		),
776		MappingException,
777		)
	725	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	726	auth_handler.complete_sso_login.assert_not_called()
	727	args = self.assertRenderedError("mapping_error")
778	728	self.assertTrue(
779		str(e.value).startswith(
	729	args[2].startswith(
780	730	"Attempted to login as '@TEST_USER_2:test' but it matches more than one user inexactly:"
781	731	)
782	732	)

787	737	store.register_user(user_id=user2.to_string(), password_hash=None)
788	738	)
789	739
790		mxid = self.get_success(
791		self.handler._map_userinfo_to_user(
792		userinfo, token, "user-agent", "10.10.10.10"
793		)
794		)
795		self.assertEqual(mxid, "@TEST_USER_2:test")
	740	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	741	auth_handler.complete_sso_login.assert_called_once_with(
	742	"@TEST_USER_2:test", ANY, ANY, None,
	743	)
796	744
797	745	def test_map_userinfo_to_invalid_localpart(self):
798	746	"""If the mapping provider generates an invalid localpart it should be rejected."""
799		userinfo = {
800		"sub": "test2",
801		"username": "föö",
802		}
803		token = {}
804
805		e = self.get_failure(
806		self.handler._map_userinfo_to_user(
807		userinfo, token, "user-agent", "10.10.10.10"
808		),
809		MappingException,
810		)
811		self.assertEqual(str(e.value), "localpart is invalid: föö")
	747	self.get_success(
	748	_make_callback_with_userinfo(self.hs, {"sub": "test2", "username": "föö"})
	749	)
	750	self.assertRenderedError("mapping_error", "localpart is invalid: föö")
812	751
813	752	@override_config(
814	753	{

821	760	)
822	761	def test_map_userinfo_to_user_retries(self):
823	762	"""The mapping provider can retry generating an MXID if the MXID is already in use."""
	763	auth_handler = self.hs.get_auth_handler()
	764	auth_handler.complete_sso_login = simple_async_mock()
	765
824	766	store = self.hs.get_datastore()
825	767	self.get_success(
826	768	store.register_user(user_id="@test_user:test", password_hash=None)

829	771	"sub": "test",
830	772	"username": "test_user",
831	773	}
832		token = {}
833		mxid = self.get_success(
834		self.handler._map_userinfo_to_user(
835		userinfo, token, "user-agent", "10.10.10.10"
836		)
837		)
	774	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	775
838	776	# test_user is already taken, so test_user1 gets registered instead.
839		self.assertEqual(mxid, "@test_user1:test")
	777	auth_handler.complete_sso_login.assert_called_once_with(
	778	"@test_user1:test", ANY, ANY, None,
	779	)
	780	auth_handler.complete_sso_login.reset_mock()
840	781
841	782	# Register all of the potential mxids for a particular OIDC username.
842	783	self.get_success(

852	793	"sub": "tester",
853	794	"username": "tester",
854	795	}
855		e = self.get_failure(
856		self.handler._map_userinfo_to_user(
857		userinfo, token, "user-agent", "10.10.10.10"
	796	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	797	auth_handler.complete_sso_login.assert_not_called()
	798	self.assertRenderedError(
	799	"mapping_error", "Unable to generate a Matrix ID from the SSO response"
	800	)
	801
	802	def test_empty_localpart(self):
	803	"""Attempts to map onto an empty localpart should be rejected."""
	804	userinfo = {
	805	"sub": "tester",
	806	"username": "",
	807	}
	808	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	809	self.assertRenderedError("mapping_error", "localpart is invalid: ")
	810
	811	@override_config(
	812	{
	813	"oidc_config": {
	814	"user_mapping_provider": {
	815	"config": {"localpart_template": "{{ user.username }}"}
	816	}
	817	}
	818	}
	819	)
	820	def test_null_localpart(self):
	821	"""Mapping onto a null localpart via an empty OIDC attribute should be rejected"""
	822	userinfo = {
	823	"sub": "tester",
	824	"username": None,
	825	}
	826	self.get_success(_make_callback_with_userinfo(self.hs, userinfo))
	827	self.assertRenderedError("mapping_error", "localpart is invalid: ")
	828
	829
	830	class UsernamePickerTestCase(HomeserverTestCase):
	831	servlets = [login.register_servlets]
	832
	833	def default_config(self):
	834	config = super().default_config()
	835	config["public_baseurl"] = BASE_URL
	836	oidc_config = {
	837	"enabled": True,
	838	"client_id": CLIENT_ID,
	839	"client_secret": CLIENT_SECRET,
	840	"issuer": ISSUER,
	841	"scopes": SCOPES,
	842	"user_mapping_provider": {
	843	"config": {"display_name_template": "{{ user.displayname }}"}
	844	},
	845	}
	846
	847	# Update this config with what's in the default config so that
	848	# override_config works as expected.
	849	oidc_config.update(config.get("oidc_config", {}))
	850	config["oidc_config"] = oidc_config
	851
	852	# whitelist this client URI so we redirect straight to it rather than
	853	# serving a confirmation page
	854	config["sso"] = {"client_whitelist": ["https://whitelisted.client"]}
	855	return config
	856
	857	def create_resource_dict(self) -> Dict[str, Resource]:
	858	d = super().create_resource_dict()
	859	d["/_synapse/client/pick_username"] = pick_username_resource(self.hs)
	860	return d
	861
	862	def test_username_picker(self):
	863	"""Test the happy path of a username picker flow."""
	864	client_redirect_url = "https://whitelisted.client"
	865
	866	# first of all, mock up an OIDC callback to the OidcHandler, which should
	867	# raise a RedirectException
	868	userinfo = {"sub": "tester", "displayname": "Jonny"}
	869	f = self.get_failure(
	870	_make_callback_with_userinfo(
	871	self.hs, userinfo, client_redirect_url=client_redirect_url
858	872	),
859		MappingException,
860		)
	873	RedirectException,
	874	)
	875
	876	# check the Location and cookies returned by the RedirectException
	877	self.assertEqual(f.value.location, b"/_synapse/client/pick_username")
	878	cookieheader = f.value.cookies[0]
	879	regex = re.compile(b"^username_mapping_session=([a-zA-Z]+);")
	880	m = regex.search(cookieheader)
	881	if not m:
	882	self.fail("cookie header %s does not match %s" % (cookieheader, regex))
	883
	884	# introspect the sso handler a bit to check that the username mapping session
	885	# looks ok.
	886	session_id = m.group(1).decode("ascii")
	887	username_mapping_sessions = self.hs.get_sso_handler()._username_mapping_sessions
	888	self.assertIn(
	889	session_id, username_mapping_sessions, "session id not found in map"
	890	)
	891	session = username_mapping_sessions[session_id]
	892	self.assertEqual(session.remote_user_id, "tester")
	893	self.assertEqual(session.display_name, "Jonny")
	894	self.assertEqual(session.client_redirect_url, client_redirect_url)
	895
	896	# the expiry time should be about 15 minutes away
	897	expected_expiry = self.clock.time_msec() + (15 * 60 * 1000)
	898	self.assertApproximates(session.expiry_time_ms, expected_expiry, tolerance=1000)
	899
	900	# Now, submit a username to the username picker, which should serve a redirect
	901	# back to the client
	902	submit_path = f.value.location + b"/submit"
	903	content = urlencode({b"username": b"bobby"}).encode("utf8")
	904	chan = self.make_request(
	905	"POST",
	906	path=submit_path,
	907	content=content,
	908	content_is_form=True,
	909	custom_headers=[
	910	("Cookie", cookieheader),
	911	# old versions of twisted don't do form-parsing without a valid
	912	# content-length header.
	913	("Content-Length", str(len(content))),
	914	],
	915	)
	916	self.assertEqual(chan.code, 302, chan.result)
	917	location_headers = chan.headers.getRawHeaders("Location")
	918	# ensure that the returned location starts with the requested redirect URL
861	919	self.assertEqual(
862		str(e.value), "Unable to generate a Matrix ID from the SSO response"
863		)
	920	location_headers[0][: len(client_redirect_url)], client_redirect_url
	921	)
	922
	923	# fish the login token out of the returned redirect uri
	924	parts = urlparse(location_headers[0])
	925	query = parse_qs(parts.query)
	926	login_token = query["loginToken"][0]
	927
	928	# finally, submit the matrix login token to the login API, which gives us our
	929	# matrix access token, mxid, and device id.
	930	chan = self.make_request(
	931	"POST", "/login", content={"type": "m.login.token", "token": login_token},
	932	)
	933	self.assertEqual(chan.code, 200, chan.result)
	934	self.assertEqual(chan.json_body["user_id"], "@bobby:test")
	935
	936
	937	async def _make_callback_with_userinfo(
	938	hs: HomeServer, userinfo: dict, client_redirect_url: str = "http://client/redirect"
	939	) -> None:
	940	"""Mock up an OIDC callback with the given userinfo dict
	941
	942	We'll pull out the OIDC handler from the homeserver, stub out a couple of methods,
	943	and poke in the userinfo dict as if it were the response to an OIDC userinfo call.
	944
	945	Args:
	946	hs: the HomeServer impl to send the callback to.
	947	userinfo: the OIDC userinfo dict
	948	client_redirect_url: the URL to redirect to on success.
	949	"""
	950	handler = hs.get_oidc_handler()
	951	handler._exchange_code = simple_async_mock(return_value={})
	952	handler._parse_id_token = simple_async_mock(return_value=userinfo)
	953	handler._fetch_userinfo = simple_async_mock(return_value=userinfo)
	954
	955	state = "state"
	956	session = handler._generate_oidc_session_token(
	957	state=state,
	958	nonce="nonce",
	959	client_redirect_url=client_redirect_url,
	960	ui_auth_session_id=None,
	961	)
	962	request = _build_callback_request("code", state, session)
	963
	964	await handler.handle_oidc_callback(request)
	965
	966
	967	def _build_callback_request(
	968	code: str,
	969	state: str,
	970	session: str,
	971	user_agent: str = "Browser",
	972	ip_address: str = "10.0.0.1",
	973	):
	974	"""Builds a fake SynapseRequest to mock the browser callback
	975
	976	Returns a Mock object which looks like the SynapseRequest we get from a browser
	977	after SSO (before we return to the client)
	978
	979	Args:
	980	code: the authorization code which would have been returned by the OIDC
	981	provider
	982	state: the "state" param which would have been passed around in the
	983	query param. Should be the same as was embedded in the session in
	984	_build_oidc_session.
	985	session: the "session" which would have been passed around in the cookie.
	986	user_agent: the user-agent to present
	987	ip_address: the IP address to pretend the request came from
	988	"""
	989	request = Mock(
	990	spec=[
	991	"args",
	992	"getCookie",
	993	"addCookie",
	994	"requestHeaders",
	995	"getClientIP",
	996	"get_user_agent",
	997	]
	998	)
	999
	1000	request.getCookie.return_value = session
	1001	request.args = {}
	1002	request.args[b"code"] = [code.encode("utf-8")]
	1003	request.args[b"state"] = [state.encode("utf-8")]
	1004	request.getClientIP.return_value = ip_address
	1005	request.get_user_agent.return_value = user_agent
	1006	return request

+26

-3

tests/handlers/test_password_providers.py less more

431	431
432	432	@override_config(
433	433	{
	434	**providers_config(CustomAuthProvider),
	435	"password_config": {"enabled": False, "localdb_enabled": False},
	436	}
	437	)
	438	def test_custom_auth_password_disabled_localdb_enabled(self):
	439	"""Check the localdb_enabled == enabled == False
	440
	441	Regression test for https://github.com/matrix-org/synapse/issues/8914: check
	442	that setting both `localdb_enabled` and `password: enabled` to False doesn't
	443	cause an exception.
	444	"""
	445	self.register_user("localuser", "localpass")
	446
	447	flows = self._get_login_flows()
	448	self.assertEqual(flows, [{"type": "test.login_type"}] + ADDITIONAL_LOGIN_FLOWS)
	449
	450	# login shouldn't work and should be rejected with a 400 ("unknown login type")
	451	channel = self._send_password_login("localuser", "localpass")
	452	self.assertEqual(channel.code, 400, channel.result)
	453	mock_password_provider.check_auth.assert_not_called()
	454
	455	@override_config(
	456	{
434	457	**providers_config(PasswordCustomAuthProvider),
435	458	"password_config": {"enabled": False},
436	459	}

527	550	self.assertEqual(channel.code, 400, channel.result)
528	551
529	552	def _get_login_flows(self) -> JsonDict:
530		_, channel = self.make_request("GET", "/_matrix/client/r0/login")
	553	channel = self.make_request("GET", "/_matrix/client/r0/login")
531	554	self.assertEqual(channel.code, 200, channel.result)
532	555	return channel.json_body["flows"]
533	556

536	559
537	560	def _send_login(self, type, user, **params) -> FakeChannel:
538	561	params.update({"identifier": {"type": "m.id.user", "user": user}, "type": type})
539		_, channel = self.make_request("POST", "/_matrix/client/r0/login", params)
	562	channel = self.make_request("POST", "/_matrix/client/r0/login", params)
540	563	return channel
541	564
542	565	def _start_delete_device_session(self, access_token, device_id) -> str:

573	596	self, access_token: str, device: str, body: Union[JsonDict, bytes] = b"",
574	597	) -> FakeChannel:
575	598	"""Delete an individual device."""
576		_, channel = self.make_request(
	599	channel = self.make_request(
577	600	"DELETE", "devices/" + device, body, access_token=access_token
578	601	)
579	602	return channel

+1

-1

tests/handlers/test_presence.py less more

462	462
463	463	def make_homeserver(self, reactor, clock):
464	464	hs = self.setup_test_homeserver(
465		"server", http_client=None, federation_sender=Mock()
	465	"server", federation_http_client=None, federation_sender=Mock()
466	466	)
467	467	return hs
468	468

+0

-2

tests/handlers/test_profile.py less more

43	43
44	44	hs = yield setup_test_homeserver(
45	45	self.addCleanup,
46		http_client=None,
47		resource_for_federation=Mock(),
48	46	federation_client=self.mock_federation,
49	47	federation_server=Mock(),
50	48	federation_registry=self.mock_registry,

+120

-51

tests/handlers/test_saml.py less more

11	11	# See the License for the specific language governing permissions and
12	12	# limitations under the License.
13	13
	14	from typing import Optional
	15
	16	from mock import Mock
	17
14	18	import attr
15	19
16	20	from synapse.api.errors import RedirectException
17		from synapse.handlers.sso import MappingException
18
	21
	22	from tests.test_utils import simple_async_mock
19	23	from tests.unittest import HomeserverTestCase, override_config
	24
	25	# Check if we have the dependencies to run the tests.
	26	try:
	27	import saml2.config
	28	from saml2.sigver import SigverError
	29
	30	has_saml2 = True
	31
	32	# pysaml2 can be installed and imported, but might not be able to find xmlsec1.
	33	config = saml2.config.SPConfig()
	34	try:
	35	config.load({"metadata": {}})
	36	has_xmlsec1 = True
	37	except SigverError:
	38	has_xmlsec1 = False
	39	except ImportError:
	40	has_saml2 = False
	41	has_xmlsec1 = False
20	42
21	43	# These are a few constants that are used as config parameters in the tests.
22	44	BASE_URL = "https://synapse/"

25	47	@attr.s
26	48	class FakeAuthnResponse:
27	49	ava = attr.ib(type=dict)
	50	assertions = attr.ib(type=list, factory=list)
	51	in_response_to = attr.ib(type=Optional[str], default=None)
28	52
29	53
30	54	class TestMappingProvider:

85	109
86	110	return hs
87	111
	112	if not has_saml2:
	113	skip = "Requires pysaml2"
	114	elif not has_xmlsec1:
	115	skip = "Requires xmlsec1"
	116
88	117	def test_map_saml_response_to_user(self):
89	118	"""Ensure that mapping the SAML response returned from a provider to an MXID works properly."""
	119
	120	# stub out the auth handler
	121	auth_handler = self.hs.get_auth_handler()
	122	auth_handler.complete_sso_login = simple_async_mock()
	123
	124	# send a mocked-up SAML response to the callback
90	125	saml_response = FakeAuthnResponse({"uid": "test_user", "username": "test_user"})
91		# The redirect_url doesn't matter with the default user mapping provider.
92		redirect_url = ""
93		mxid = self.get_success(
94		self.handler._map_saml_response_to_user(
95		saml_response, redirect_url, "user-agent", "10.10.10.10"
96		)
97		)
98		self.assertEqual(mxid, "@test_user:test")
	126	request = _mock_request()
	127	self.get_success(
	128	self.handler._handle_authn_response(request, saml_response, "redirect_uri")
	129	)
	130
	131	# check that the auth handler got called as expected
	132	auth_handler.complete_sso_login.assert_called_once_with(
	133	"@test_user:test", request, "redirect_uri", None
	134	)
99	135
100	136	@override_config({"saml2_config": {"grandfathered_mxid_source_attribute": "mxid"}})
101	137	def test_map_saml_response_to_existing_user(self):

105	141	store.register_user(user_id="@test_user:test", password_hash=None)
106	142	)
107	143
	144	# stub out the auth handler
	145	auth_handler = self.hs.get_auth_handler()
	146	auth_handler.complete_sso_login = simple_async_mock()
	147
108	148	# Map a user via SSO.
109	149	saml_response = FakeAuthnResponse(
110	150	{"uid": "tester", "mxid": ["test_user"], "username": "test_user"}
111	151	)
112		redirect_url = ""
113		mxid = self.get_success(
114		self.handler._map_saml_response_to_user(
115		saml_response, redirect_url, "user-agent", "10.10.10.10"
116		)
117		)
118		self.assertEqual(mxid, "@test_user:test")
	152	request = _mock_request()
	153	self.get_success(
	154	self.handler._handle_authn_response(request, saml_response, "")
	155	)
	156
	157	# check that the auth handler got called as expected
	158	auth_handler.complete_sso_login.assert_called_once_with(
	159	"@test_user:test", request, "", None
	160	)
119	161
120	162	# Subsequent calls should map to the same mxid.
121		mxid = self.get_success(
122		self.handler._map_saml_response_to_user(
123		saml_response, redirect_url, "user-agent", "10.10.10.10"
124		)
125		)
126		self.assertEqual(mxid, "@test_user:test")
	163	auth_handler.complete_sso_login.reset_mock()
	164	self.get_success(
	165	self.handler._handle_authn_response(request, saml_response, "")
	166	)
	167	auth_handler.complete_sso_login.assert_called_once_with(
	168	"@test_user:test", request, "", None
	169	)
127	170
128	171	def test_map_saml_response_to_invalid_localpart(self):
129	172	"""If the mapping provider generates an invalid localpart it should be rejected."""
	173
	174	# stub out the auth handler
	175	auth_handler = self.hs.get_auth_handler()
	176	auth_handler.complete_sso_login = simple_async_mock()
	177
	178	# mock out the error renderer too
	179	sso_handler = self.hs.get_sso_handler()
	180	sso_handler.render_error = Mock(return_value=None)
	181
130	182	saml_response = FakeAuthnResponse({"uid": "test", "username": "föö"})
131		redirect_url = ""
132		e = self.get_failure(
133		self.handler._map_saml_response_to_user(
134		saml_response, redirect_url, "user-agent", "10.10.10.10"
135		),
136		MappingException,
137		)
138		self.assertEqual(str(e.value), "localpart is invalid: föö")
	183	request = _mock_request()
	184	self.get_success(
	185	self.handler._handle_authn_response(request, saml_response, ""),
	186	)
	187	sso_handler.render_error.assert_called_once_with(
	188	request, "mapping_error", "localpart is invalid: föö"
	189	)
	190	auth_handler.complete_sso_login.assert_not_called()
139	191
140	192	def test_map_saml_response_to_user_retries(self):
141	193	"""The mapping provider can retry generating an MXID if the MXID is already in use."""
	194
	195	# stub out the auth handler and error renderer
	196	auth_handler = self.hs.get_auth_handler()
	197	auth_handler.complete_sso_login = simple_async_mock()
	198	sso_handler = self.hs.get_sso_handler()
	199	sso_handler.render_error = Mock(return_value=None)
	200
	201	# register a user to occupy the first-choice MXID
142	202	store = self.hs.get_datastore()
143	203	self.get_success(
144	204	store.register_user(user_id="@test_user:test", password_hash=None)
145	205	)
	206
	207	# send the fake SAML response
146	208	saml_response = FakeAuthnResponse({"uid": "test", "username": "test_user"})
147		redirect_url = ""
148		mxid = self.get_success(
149		self.handler._map_saml_response_to_user(
150		saml_response, redirect_url, "user-agent", "10.10.10.10"
151		)
152		)
	209	request = _mock_request()
	210	self.get_success(
	211	self.handler._handle_authn_response(request, saml_response, ""),
	212	)
	213
153	214	# test_user is already taken, so test_user1 gets registered instead.
154		self.assertEqual(mxid, "@test_user1:test")
	215	auth_handler.complete_sso_login.assert_called_once_with(
	216	"@test_user1:test", request, "", None
	217	)
	218	auth_handler.complete_sso_login.reset_mock()
155	219
156	220	# Register all of the potential mxids for a particular SAML username.
157	221	self.get_success(

164	228
165	229	# Now attempt to map to a username, this will fail since all potential usernames are taken.
166	230	saml_response = FakeAuthnResponse({"uid": "tester", "username": "tester"})
167		e = self.get_failure(
168		self.handler._map_saml_response_to_user(
169		saml_response, redirect_url, "user-agent", "10.10.10.10"
170		),
171		MappingException,
172		)
173		self.assertEqual(
174		str(e.value), "Unable to generate a Matrix ID from the SSO response"
175		)
	231	self.get_success(
	232	self.handler._handle_authn_response(request, saml_response, ""),
	233	)
	234	sso_handler.render_error.assert_called_once_with(
	235	request,
	236	"mapping_error",
	237	"Unable to generate a Matrix ID from the SSO response",
	238	)
	239	auth_handler.complete_sso_login.assert_not_called()
176	240
177	241	@override_config(
178	242	{

184	248	}
185	249	)
186	250	def test_map_saml_response_redirect(self):
	251	"""Test a mapping provider that raises a RedirectException"""
	252
187	253	saml_response = FakeAuthnResponse({"uid": "test", "username": "test_user"})
188		redirect_url = ""
	254	request = _mock_request()
189	255	e = self.get_failure(
190		self.handler._map_saml_response_to_user(
191		saml_response, redirect_url, "user-agent", "10.10.10.10"
192		),
	256	self.handler._handle_authn_response(request, saml_response, ""),
193	257	RedirectException,
194	258	)
195	259	self.assertEqual(e.value.location, b"https://custom-saml-redirect/")
	260
	261
	262	def _mock_request():
	263	"""Returns a mock which will stand in as a SynapseRequest"""
	264	return Mock(spec=["getClientIP", "get_user_agent"])

+12

-7

tests/handlers/test_typing.py less more

14	14
15	15
16	16	import json
	17	from typing import Dict
17	18
18	19	from mock import ANY, Mock, call
19	20
20	21	from twisted.internet import defer
	22	from twisted.web.resource import Resource
21	23
22	24	from synapse.api.errors import AuthError
	25	from synapse.federation.transport.server import TransportLayerServer
23	26	from synapse.types import UserID, create_requester
24	27
25	28	from tests import unittest
26	29	from tests.test_utils import make_awaitable
27	30	from tests.unittest import override_config
28		from tests.utils import register_federation_servlets
29	31
30	32	# Some local users to test with
31	33	U_APPLE = UserID.from_string("@apple:test")

52	54
53	55
54	56	class TypingNotificationsTestCase(unittest.HomeserverTestCase):
55		servlets = [register_federation_servlets]
56
57	57	def make_homeserver(self, reactor, clock):
58	58	# we mock out the keyring so as to skip the authentication check on the
59	59	# federation API call.

69	69
70	70	hs = self.setup_test_homeserver(
71	71	notifier=Mock(),
72		http_client=mock_federation_client,
	72	federation_http_client=mock_federation_client,
73	73	keyring=mock_keyring,
74	74	replication_streams={},
75	75	)
76	76
77	77	return hs
	78
	79	def create_resource_dict(self) -> Dict[str, Resource]:
	80	d = super().create_resource_dict()
	81	d["/_matrix/federation"] = TransportLayerServer(self.hs)
	82	return d
78	83
79	84	def prepare(self, reactor, clock, hs):
80	85	mock_notifier = hs.get_notifier()

191	196	)
192	197	)
193	198
194		put_json = self.hs.get_http_client().put_json
	199	put_json = self.hs.get_federation_http_client().put_json
195	200	put_json.assert_called_once_with(
196	201	"farm",
197	202	path="/_matrix/federation/v1/send/1000000",

214	219
215	220	self.assertEquals(self.event_source.get_current_key(), 0)
216	221
217		(request, channel) = self.make_request(
	222	channel = self.make_request(
218	223	"PUT",
219	224	"/_matrix/federation/v1/send/1000000",
220	225	_make_edu_transaction_json(

269	274
270	275	self.on_new_event.assert_has_calls([call("typing_key", 1, rooms=[ROOM_ID])])
271	276
272		put_json = self.hs.get_http_client().put_json
	277	put_json = self.hs.get_federation_http_client().put_json
273	278	put_json.assert_called_once_with(
274	279	"farm",
275	280	path="/_matrix/federation/v1/send/1000000",

+43

-5

tests/handlers/test_user_directory.py less more

53	53	user_id=support_user_id, password_hash=None, user_type=UserTypes.SUPPORT
54	54	)
55	55	)
	56	regular_user_id = "@regular:test"
	57	self.get_success(
	58	self.store.register_user(user_id=regular_user_id, password_hash=None)
	59	)
56	60
57	61	self.get_success(
58	62	self.handler.handle_local_profile_change(support_user_id, None)

62	66	display_name = "display_name"
63	67
64	68	profile_info = ProfileInfo(avatar_url="avatar_url", display_name=display_name)
65		regular_user_id = "@regular:test"
66	69	self.get_success(
67	70	self.handler.handle_local_profile_change(regular_user_id, profile_info)
68	71	)
69	72	profile = self.get_success(self.store.get_user_in_directory(regular_user_id))
70	73	self.assertTrue(profile["display_name"] == display_name)
	74
	75	def test_handle_local_profile_change_with_deactivated_user(self):
	76	# create user
	77	r_user_id = "@regular:test"
	78	self.get_success(
	79	self.store.register_user(user_id=r_user_id, password_hash=None)
	80	)
	81
	82	# update profile
	83	display_name = "Regular User"
	84	profile_info = ProfileInfo(avatar_url="avatar_url", display_name=display_name)
	85	self.get_success(
	86	self.handler.handle_local_profile_change(r_user_id, profile_info)
	87	)
	88
	89	# profile is in directory
	90	profile = self.get_success(self.store.get_user_in_directory(r_user_id))
	91	self.assertTrue(profile["display_name"] == display_name)
	92
	93	# deactivate user
	94	self.get_success(self.store.set_user_deactivated_status(r_user_id, True))
	95	self.get_success(self.handler.handle_user_deactivated(r_user_id))
	96
	97	# profile is not in directory
	98	profile = self.get_success(self.store.get_user_in_directory(r_user_id))
	99	self.assertTrue(profile is None)
	100
	101	# update profile after deactivation
	102	self.get_success(
	103	self.handler.handle_local_profile_change(r_user_id, profile_info)
	104	)
	105
	106	# profile is furthermore not in directory
	107	profile = self.get_success(self.store.get_user_in_directory(r_user_id))
	108	self.assertTrue(profile is None)
71	109
72	110	def test_handle_user_deactivated_support_user(self):
73	111	s_user_id = "@support:test"

269	307	spam_checker = self.hs.get_spam_checker()
270	308
271	309	class AllowAll:
272		def check_username_for_spam(self, user_profile):
	310	async def check_username_for_spam(self, user_profile):
273	311	# Allow all users.
274	312	return False
275	313

282	320
283	321	# Configure a spam checker that filters all users.
284	322	class BlockAll:
285		def check_username_for_spam(self, user_profile):
	323	async def check_username_for_spam(self, user_profile):
286	324	# All users are spammy.
287	325	return True
288	326

533	571	self.helper.join(room, user=u2)
534	572
535	573	# Assert user directory is not empty
536		request, channel = self.make_request(
	574	channel = self.make_request(
537	575	"POST", b"user_directory/search", b'{"search_term":"user2"}'
538	576	)
539	577	self.assertEquals(200, channel.code, channel.result)

541	579
542	580	# Disable user directory and check search returns nothing
543	581	self.config.user_directory_search_enabled = False
544		request, channel = self.make_request(
	582	channel = self.make_request(
545	583	"POST", b"user_directory/search", b'{"search_term":"user2"}'
546	584	)
547	585	self.assertEquals(200, channel.code, channel.result)

+30

-0

tests/http/federation/test_matrix_federation_agent.py less more

16	16	from mock import Mock
17	17
18	18	import treq
	19	from netaddr import IPSet
19	20	from service_identity import VerificationError
20	21	from zope.interface import implementer
21	22

34	35	from synapse.http.federation.matrix_federation_agent import MatrixFederationAgent
35	36	from synapse.http.federation.srv_resolver import Server
36	37	from synapse.http.federation.well_known_resolver import (
	38	WELL_KNOWN_MAX_SIZE,
37	39	WellKnownResolver,
38	40	_cache_period_from_headers,
39	41	)

102	104	reactor=self.reactor,
103	105	tls_client_options_factory=self.tls_factory,
104	106	user_agent="test-agent", # Note that this is unused since _well_known_resolver is provided.
	107	ip_blacklist=IPSet(),
105	108	_srv_resolver=self.mock_resolver,
106	109	_well_known_resolver=self.well_known_resolver,
107	110	)

735	738	reactor=self.reactor,
736	739	tls_client_options_factory=tls_factory,
737	740	user_agent=b"test-agent", # This is unused since _well_known_resolver is passed below.
	741	ip_blacklist=IPSet(),
738	742	_srv_resolver=self.mock_resolver,
739	743	_well_known_resolver=WellKnownResolver(
740	744	self.reactor,

1102	1106
1103	1107	r = self.successResultOf(fetch_d)
1104	1108	self.assertEqual(r.delegated_server, None)
	1109
	1110	def test_well_known_too_large(self):
	1111	"""A well-known query that returns a result which is too large should be rejected."""
	1112	self.reactor.lookups["testserv"] = "1.2.3.4"
	1113
	1114	fetch_d = defer.ensureDeferred(
	1115	self.well_known_resolver.get_well_known(b"testserv")
	1116	)
	1117
	1118	# there should be an attempt to connect on port 443 for the .well-known
	1119	clients = self.reactor.tcpClients
	1120	self.assertEqual(len(clients), 1)
	1121	(host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)
	1122	self.assertEqual(host, "1.2.3.4")
	1123	self.assertEqual(port, 443)
	1124
	1125	self._handle_well_known_connection(
	1126	client_factory,
	1127	expected_sni=b"testserv",
	1128	response_headers={b"Cache-Control": b"max-age=1000"},
	1129	content=b'{ "m.server": "' + (b"a" * WELL_KNOWN_MAX_SIZE) + b'" }',
	1130	)
	1131
	1132	# The result is sucessful, but disabled delegation.
	1133	r = self.successResultOf(fetch_d)
	1134	self.assertIsNone(r.delegated_server)
1105	1135
1106	1136	def test_srv_fallbacks(self):
1107	1137	"""Test that other SRV results are tried if the first one fails.

+4

-4

tests/http/test_additional_resource.py less more

45	45	handler = _AsyncTestCustomEndpoint({}, None).handle_request
46	46	resource = AdditionalResource(self.hs, handler)
47	47
48		request, channel = make_request(self.reactor, FakeSite(resource), "GET", "/")
	48	channel = make_request(self.reactor, FakeSite(resource), "GET", "/")
49	49
50		self.assertEqual(request.code, 200)
	50	self.assertEqual(channel.code, 200)
51	51	self.assertEqual(channel.json_body, {"some_key": "some_value_async"})
52	52
53	53	def test_sync(self):
54	54	handler = _SyncTestCustomEndpoint({}, None).handle_request
55	55	resource = AdditionalResource(self.hs, handler)
56	56
57		request, channel = make_request(self.reactor, FakeSite(resource), "GET", "/")
	57	channel = make_request(self.reactor, FakeSite(resource), "GET", "/")
58	58
59		self.assertEqual(request.code, 200)
	59	self.assertEqual(channel.code, 200)
60	60	self.assertEqual(channel.json_body, {"some_key": "some_value_sync"})

+130

-0

tests/http/test_proxyagent.py less more

14	14	import logging
15	15
16	16	import treq
	17	from netaddr import IPSet
17	18
18	19	from twisted.internet import interfaces # noqa: F401
19	20	from twisted.internet.protocol import Factory
20	21	from twisted.protocols.tls import TLSMemoryBIOFactory
21	22	from twisted.web.http import HTTPChannel
22	23
	24	from synapse.http.client import BlacklistingReactorWrapper
23	25	from synapse.http.proxyagent import ProxyAgent
24	26
25	27	from tests.http import TestServerTLSConnectionFactory, get_test_https_policy

291	293	body = self.successResultOf(treq.content(resp))
292	294	self.assertEqual(body, b"result")
293	295
	296	def test_http_request_via_proxy_with_blacklist(self):
	297	# The blacklist includes the configured proxy IP.
	298	agent = ProxyAgent(
	299	BlacklistingReactorWrapper(
	300	self.reactor, ip_whitelist=None, ip_blacklist=IPSet(["1.0.0.0/8"])
	301	),
	302	self.reactor,
	303	http_proxy=b"proxy.com:8888",
	304	)
	305
	306	self.reactor.lookups["proxy.com"] = "1.2.3.5"
	307	d = agent.request(b"GET", b"http://test.com")
	308
	309	# there should be a pending TCP connection
	310	clients = self.reactor.tcpClients
	311	self.assertEqual(len(clients), 1)
	312	(host, port, client_factory, _timeout, _bindAddress) = clients[0]
	313	self.assertEqual(host, "1.2.3.5")
	314	self.assertEqual(port, 8888)
	315
	316	# make a test server, and wire up the client
	317	http_server = self._make_connection(
	318	client_factory, _get_test_protocol_factory()
	319	)
	320
	321	# the FakeTransport is async, so we need to pump the reactor
	322	self.reactor.advance(0)
	323
	324	# now there should be a pending request
	325	self.assertEqual(len(http_server.requests), 1)
	326
	327	request = http_server.requests[0]
	328	self.assertEqual(request.method, b"GET")
	329	self.assertEqual(request.path, b"http://test.com")
	330	self.assertEqual(request.requestHeaders.getRawHeaders(b"host"), [b"test.com"])
	331	request.write(b"result")
	332	request.finish()
	333
	334	self.reactor.advance(0)
	335
	336	resp = self.successResultOf(d)
	337	body = self.successResultOf(treq.content(resp))
	338	self.assertEqual(body, b"result")
	339
	340	def test_https_request_via_proxy_with_blacklist(self):
	341	# The blacklist includes the configured proxy IP.
	342	agent = ProxyAgent(
	343	BlacklistingReactorWrapper(
	344	self.reactor, ip_whitelist=None, ip_blacklist=IPSet(["1.0.0.0/8"])
	345	),
	346	self.reactor,
	347	contextFactory=get_test_https_policy(),
	348	https_proxy=b"proxy.com",
	349	)
	350
	351	self.reactor.lookups["proxy.com"] = "1.2.3.5"
	352	d = agent.request(b"GET", b"https://test.com/abc")
	353
	354	# there should be a pending TCP connection
	355	clients = self.reactor.tcpClients
	356	self.assertEqual(len(clients), 1)
	357	(host, port, client_factory, _timeout, _bindAddress) = clients[0]
	358	self.assertEqual(host, "1.2.3.5")
	359	self.assertEqual(port, 1080)
	360
	361	# make a test HTTP server, and wire up the client
	362	proxy_server = self._make_connection(
	363	client_factory, _get_test_protocol_factory()
	364	)
	365
	366	# fish the transports back out so that we can do the old switcheroo
	367	s2c_transport = proxy_server.transport
	368	client_protocol = s2c_transport.other
	369	c2s_transport = client_protocol.transport
	370
	371	# the FakeTransport is async, so we need to pump the reactor
	372	self.reactor.advance(0)
	373
	374	# now there should be a pending CONNECT request
	375	self.assertEqual(len(proxy_server.requests), 1)
	376
	377	request = proxy_server.requests[0]
	378	self.assertEqual(request.method, b"CONNECT")
	379	self.assertEqual(request.path, b"test.com:443")
	380
	381	# tell the proxy server not to close the connection
	382	proxy_server.persistent = True
	383
	384	# this just stops the http Request trying to do a chunked response
	385	# request.setHeader(b"Content-Length", b"0")
	386	request.finish()
	387
	388	# now we can replace the proxy channel with a new, SSL-wrapped HTTP channel
	389	ssl_factory = _wrap_server_factory_for_tls(_get_test_protocol_factory())
	390	ssl_protocol = ssl_factory.buildProtocol(None)
	391	http_server = ssl_protocol.wrappedProtocol
	392
	393	ssl_protocol.makeConnection(
	394	FakeTransport(client_protocol, self.reactor, ssl_protocol)
	395	)
	396	c2s_transport.other = ssl_protocol
	397
	398	self.reactor.advance(0)
	399
	400	server_name = ssl_protocol._tlsConnection.get_servername()
	401	expected_sni = b"test.com"
	402	self.assertEqual(
	403	server_name,
	404	expected_sni,
	405	"Expected SNI %s but got %s" % (expected_sni, server_name),
	406	)
	407
	408	# now there should be a pending request
	409	self.assertEqual(len(http_server.requests), 1)
	410
	411	request = http_server.requests[0]
	412	self.assertEqual(request.method, b"GET")
	413	self.assertEqual(request.path, b"/abc")
	414	self.assertEqual(request.requestHeaders.getRawHeaders(b"host"), [b"test.com"])
	415	request.write(b"result")
	416	request.finish()
	417
	418	self.reactor.advance(0)
	419
	420	resp = self.successResultOf(d)
	421	body = self.successResultOf(treq.content(resp))
	422	self.assertEqual(body, b"result")
	423
294	424
295	425	def _wrap_server_factory_for_tls(factory, sanlist=None):
296	426	"""Wrap an existing Protocol Factory with a test TLSMemoryBIOFactory

+43

-27

tests/logging/test_terse_json.py less more

17	17	from io import StringIO
18	18
19	19	from synapse.logging._terse_json import JsonFormatter, TerseJsonFormatter
	20	from synapse.logging.context import LoggingContext, LoggingContextFilter
20	21
21	22	from tests.logging import LoggerCleanupMixin
22	23	from tests.unittest import TestCase
23	24
24	25
25	26	class TerseJsonTestCase(LoggerCleanupMixin, TestCase):
	27	def setUp(self):
	28	self.output = StringIO()
	29
	30	def get_log_line(self):
	31	# One log message, with a single trailing newline.
	32	data = self.output.getvalue()
	33	logs = data.splitlines()
	34	self.assertEqual(len(logs), 1)
	35	self.assertEqual(data.count("\n"), 1)
	36	return json.loads(logs[0])
	37
26	38	def test_terse_json_output(self):
27	39	"""
28	40	The Terse JSON formatter converts log messages to JSON.
29	41	"""
30		output = StringIO()
31
32		handler = logging.StreamHandler(output)
	42	handler = logging.StreamHandler(self.output)
33	43	handler.setFormatter(TerseJsonFormatter())
34	44	logger = self.get_logger(handler)
35	45
36	46	logger.info("Hello there, %s!", "wally")
37	47
38		# One log message, with a single trailing newline.
39		data = output.getvalue()
40		logs = data.splitlines()
41		self.assertEqual(len(logs), 1)
42		self.assertEqual(data.count("\n"), 1)
43		log = json.loads(logs[0])
	48	log = self.get_log_line()
44	49
45	50	# The terse logger should give us these keys.
46	51	expected_log_keys = [

56	61	"""
57	62	Additional information can be included in the structured logging.
58	63	"""
59		output = StringIO()
60
61		handler = logging.StreamHandler(output)
	64	handler = logging.StreamHandler(self.output)
62	65	handler.setFormatter(TerseJsonFormatter())
63	66	logger = self.get_logger(handler)
64	67

66	69	"Hello there, %s!", "wally", extra={"foo": "bar", "int": 3, "bool": True}
67	70	)
68	71
69		# One log message, with a single trailing newline.
70		data = output.getvalue()
71		logs = data.splitlines()
72		self.assertEqual(len(logs), 1)
73		self.assertEqual(data.count("\n"), 1)
74		log = json.loads(logs[0])
	72	log = self.get_log_line()
75	73
76	74	# The terse logger should give us these keys.
77	75	expected_log_keys = [

95	93	"""
96	94	The Terse JSON formatter converts log messages to JSON.
97	95	"""
98		output = StringIO()
99
100		handler = logging.StreamHandler(output)
	96	handler = logging.StreamHandler(self.output)
101	97	handler.setFormatter(JsonFormatter())
102	98	logger = self.get_logger(handler)
103	99
104	100	logger.info("Hello there, %s!", "wally")
105	101
106		# One log message, with a single trailing newline.
107		data = output.getvalue()
108		logs = data.splitlines()
109		self.assertEqual(len(logs), 1)
110		self.assertEqual(data.count("\n"), 1)
111		log = json.loads(logs[0])
	102	log = self.get_log_line()
112	103
113	104	# The terse logger should give us these keys.
114	105	expected_log_keys = [

118	109	]
119	110	self.assertCountEqual(log.keys(), expected_log_keys)
120	111	self.assertEqual(log["log"], "Hello there, wally!")
	112
	113	def test_with_context(self):
	114	"""
	115	The logging context should be added to the JSON response.
	116	"""
	117	handler = logging.StreamHandler(self.output)
	118	handler.setFormatter(JsonFormatter())
	119	handler.addFilter(LoggingContextFilter())
	120	logger = self.get_logger(handler)
	121
	122	with LoggingContext(request="test"):
	123	logger.info("Hello there, %s!", "wally")
	124
	125	log = self.get_log_line()
	126
	127	# The terse logger should give us these keys.
	128	expected_log_keys = [
	129	"log",
	130	"level",
	131	"namespace",
	132	"request",
	133	]
	134	self.assertCountEqual(log.keys(), expected_log_keys)
	135	self.assertEqual(log["log"], "Hello there, wally!")
	136	self.assertEqual(log["request"], "test")

+3

-3

tests/push/test_email.py less more

208	208	)
209	209	pushers = list(pushers)
210	210	self.assertEqual(len(pushers), 1)
211		last_stream_ordering = pushers[0]["last_stream_ordering"]
	211	last_stream_ordering = pushers[0].last_stream_ordering
212	212
213	213	# Advance time a bit, so the pusher will register something has happened
214	214	self.pump(10)

219	219	)
220	220	pushers = list(pushers)
221	221	self.assertEqual(len(pushers), 1)
222		self.assertEqual(last_stream_ordering, pushers[0]["last_stream_ordering"])
	222	self.assertEqual(last_stream_ordering, pushers[0].last_stream_ordering)
223	223
224	224	# One email was attempted to be sent
225	225	self.assertEqual(len(self.email_attempts), 1)

237	237	)
238	238	pushers = list(pushers)
239	239	self.assertEqual(len(pushers), 1)
240		self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)
	240	self.assertTrue(pushers[0].last_stream_ordering > last_stream_ordering)

+91

-27

tests/push/test_http.py less more

17	17
18	18	import synapse.rest.admin
19	19	from synapse.logging.context import make_deferred_yieldable
	20	from synapse.push import PusherConfigException
20	21	from synapse.rest.client.v1 import login, room
21	22	from synapse.rest.client.v2_alpha import receipts
22	23

33	34	user_id = True
34	35	hijack_auth = False
35	36
	37	def default_config(self):
	38	config = super().default_config()
	39	config["start_pushers"] = True
	40	return config
	41
36	42	def make_homeserver(self, reactor, clock):
37	43	self.push_attempts = []
38	44

45	51
46	52	m.post_json_get_json = post_json_get_json
47	53
48		config = self.default_config()
49		config["start_pushers"] = True
50
51		hs = self.setup_test_homeserver(config=config, proxied_http_client=m)
	54	hs = self.setup_test_homeserver(proxied_blacklisted_http_client=m)
52	55
53	56	return hs
	57
	58	def test_invalid_configuration(self):
	59	"""Invalid push configurations should be rejected."""
	60	# Register the user who gets notified
	61	user_id = self.register_user("user", "pass")
	62	access_token = self.login("user", "pass")
	63
	64	# Register the pusher
	65	user_tuple = self.get_success(
	66	self.hs.get_datastore().get_user_by_access_token(access_token)
	67	)
	68	token_id = user_tuple.token_id
	69
	70	def test_data(data):
	71	self.get_failure(
	72	self.hs.get_pusherpool().add_pusher(
	73	user_id=user_id,
	74	access_token=token_id,
	75	kind="http",
	76	app_id="m.http",
	77	app_display_name="HTTP Push Notifications",
	78	device_display_name="pushy push",
	79	pushkey="a@example.com",
	80	lang=None,
	81	data=data,
	82	),
	83	PusherConfigException,
	84	)
	85
	86	# Data must be provided with a URL.
	87	test_data(None)
	88	test_data({})
	89	test_data({"url": 1})
	90	# A bare domain name isn't accepted.
	91	test_data({"url": "example.com"})
	92	# A URL without a path isn't accepted.
	93	test_data({"url": "http://example.com"})
	94	# A url with an incorrect path isn't accepted.
	95	test_data({"url": "http://example.com/foo"})
54	96
55	97	def test_sends_http(self):
56	98	"""

81	123	device_display_name="pushy push",
82	124	pushkey="a@example.com",
83	125	lang=None,
84		data={"url": "example.com"},
	126	data={"url": "http://example.com/_matrix/push/v1/notify"},
85	127	)
86	128	)
87	129

101	143	)
102	144	pushers = list(pushers)
103	145	self.assertEqual(len(pushers), 1)
104		last_stream_ordering = pushers[0]["last_stream_ordering"]
	146	last_stream_ordering = pushers[0].last_stream_ordering
105	147
106	148	# Advance time a bit, so the pusher will register something has happened
107	149	self.pump()

112	154	)
113	155	pushers = list(pushers)
114	156	self.assertEqual(len(pushers), 1)
115		self.assertEqual(last_stream_ordering, pushers[0]["last_stream_ordering"])
	157	self.assertEqual(last_stream_ordering, pushers[0].last_stream_ordering)
116	158
117	159	# One push was attempted to be sent -- it'll be the first message
118	160	self.assertEqual(len(self.push_attempts), 1)
119		self.assertEqual(self.push_attempts[0][1], "example.com")
	161	self.assertEqual(
	162	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	163	)
120	164	self.assertEqual(
121	165	self.push_attempts[0][2]["notification"]["content"]["body"], "Hi!"
122	166	)

131	175	)
132	176	pushers = list(pushers)
133	177	self.assertEqual(len(pushers), 1)
134		self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)
135		last_stream_ordering = pushers[0]["last_stream_ordering"]
	178	self.assertTrue(pushers[0].last_stream_ordering > last_stream_ordering)
	179	last_stream_ordering = pushers[0].last_stream_ordering
136	180
137	181	# Now it'll try and send the second push message, which will be the second one
138	182	self.assertEqual(len(self.push_attempts), 2)
139		self.assertEqual(self.push_attempts[1][1], "example.com")
	183	self.assertEqual(
	184	self.push_attempts[1][1], "http://example.com/_matrix/push/v1/notify"
	185	)
140	186	self.assertEqual(
141	187	self.push_attempts[1][2]["notification"]["content"]["body"], "There!"
142	188	)

151	197	)
152	198	pushers = list(pushers)
153	199	self.assertEqual(len(pushers), 1)
154		self.assertTrue(pushers[0]["last_stream_ordering"] > last_stream_ordering)
	200	self.assertTrue(pushers[0].last_stream_ordering > last_stream_ordering)
155	201
156	202	def test_sends_high_priority_for_encrypted(self):
157	203	"""

193	239	device_display_name="pushy push",
194	240	pushkey="a@example.com",
195	241	lang=None,
196		data={"url": "example.com"},
	242	data={"url": "http://example.com/_matrix/push/v1/notify"},
197	243	)
198	244	)
199	245

229	275
230	276	# Check our push made it with high priority
231	277	self.assertEqual(len(self.push_attempts), 1)
232		self.assertEqual(self.push_attempts[0][1], "example.com")
	278	self.assertEqual(
	279	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	280	)
233	281	self.assertEqual(self.push_attempts[0][2]["notification"]["prio"], "high")
234	282
235	283	# Add yet another person — we want to make this room not a 1:1

267	315	# Advance time a bit, so the pusher will register something has happened
268	316	self.pump()
269	317	self.assertEqual(len(self.push_attempts), 2)
270		self.assertEqual(self.push_attempts[1][1], "example.com")
	318	self.assertEqual(
	319	self.push_attempts[1][1], "http://example.com/_matrix/push/v1/notify"
	320	)
271	321	self.assertEqual(self.push_attempts[1][2]["notification"]["prio"], "high")
272	322
273	323	def test_sends_high_priority_for_one_to_one_only(self):

309	359	device_display_name="pushy push",
310	360	pushkey="a@example.com",
311	361	lang=None,
312		data={"url": "example.com"},
	362	data={"url": "http://example.com/_matrix/push/v1/notify"},
313	363	)
314	364	)
315	365

325	375
326	376	# Check our push made it with high priority — this is a one-to-one room
327	377	self.assertEqual(len(self.push_attempts), 1)
328		self.assertEqual(self.push_attempts[0][1], "example.com")
	378	self.assertEqual(
	379	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	380	)
329	381	self.assertEqual(self.push_attempts[0][2]["notification"]["prio"], "high")
330	382
331	383	# Yet another user joins

344	396	# Advance time a bit, so the pusher will register something has happened
345	397	self.pump()
346	398	self.assertEqual(len(self.push_attempts), 2)
347		self.assertEqual(self.push_attempts[1][1], "example.com")
	399	self.assertEqual(
	400	self.push_attempts[1][1], "http://example.com/_matrix/push/v1/notify"
	401	)
348	402
349	403	# check that this is low-priority
350	404	self.assertEqual(self.push_attempts[1][2]["notification"]["prio"], "low")

391	445	device_display_name="pushy push",
392	446	pushkey="a@example.com",
393	447	lang=None,
394		data={"url": "example.com"},
	448	data={"url": "http://example.com/_matrix/push/v1/notify"},
395	449	)
396	450	)
397	451

407	461
408	462	# Check our push made it with high priority
409	463	self.assertEqual(len(self.push_attempts), 1)
410		self.assertEqual(self.push_attempts[0][1], "example.com")
	464	self.assertEqual(
	465	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	466	)
411	467	self.assertEqual(self.push_attempts[0][2]["notification"]["prio"], "high")
412	468
413	469	# Send another event, this time with no mention

416	472	# Advance time a bit, so the pusher will register something has happened
417	473	self.pump()
418	474	self.assertEqual(len(self.push_attempts), 2)
419		self.assertEqual(self.push_attempts[1][1], "example.com")
	475	self.assertEqual(
	476	self.push_attempts[1][1], "http://example.com/_matrix/push/v1/notify"
	477	)
420	478
421	479	# check that this is low-priority
422	480	self.assertEqual(self.push_attempts[1][2]["notification"]["prio"], "low")

464	522	device_display_name="pushy push",
465	523	pushkey="a@example.com",
466	524	lang=None,
467		data={"url": "example.com"},
	525	data={"url": "http://example.com/_matrix/push/v1/notify"},
468	526	)
469	527	)
470	528

484	542
485	543	# Check our push made it with high priority
486	544	self.assertEqual(len(self.push_attempts), 1)
487		self.assertEqual(self.push_attempts[0][1], "example.com")
	545	self.assertEqual(
	546	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	547	)
488	548	self.assertEqual(self.push_attempts[0][2]["notification"]["prio"], "high")
489	549
490	550	# Send another event, this time as someone without the power of @room

495	555	# Advance time a bit, so the pusher will register something has happened
496	556	self.pump()
497	557	self.assertEqual(len(self.push_attempts), 2)
498		self.assertEqual(self.push_attempts[1][1], "example.com")
	558	self.assertEqual(
	559	self.push_attempts[1][1], "http://example.com/_matrix/push/v1/notify"
	560	)
499	561
500	562	# check that this is low-priority
501	563	self.assertEqual(self.push_attempts[1][2]["notification"]["prio"], "low")

569	631	device_display_name="pushy push",
570	632	pushkey="a@example.com",
571	633	lang=None,
572		data={"url": "example.com"},
	634	data={"url": "http://example.com/_matrix/push/v1/notify"},
573	635	)
574	636	)
575	637

588	650
589	651	# Check our push made it
590	652	self.assertEqual(len(self.push_attempts), 1)
591		self.assertEqual(self.push_attempts[0][1], "example.com")
	653	self.assertEqual(
	654	self.push_attempts[0][1], "http://example.com/_matrix/push/v1/notify"
	655	)
592	656
593	657	# Check that the unread count for the room is 0
594	658	#

602	666	# This will actually trigger a new notification to be sent out so that
603	667	# even if the user does not receive another message, their unread
604	668	# count goes down
605		request, channel = self.make_request(
	669	channel = self.make_request(
606	670	"POST",
607	671	"/rooms/%s/receipt/m.read/%s" % (room_id, first_message_event_id),
608	672	{},

+10

-8

tests/replication/_base.py less more

12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14	14	import logging
15		from typing import Any, Callable, List, Optional, Tuple
	15	from typing import Any, Callable, Dict, List, Optional, Tuple
16	16
17	17	import attr
18	18

20	20	from twisted.internet.protocol import Protocol
21	21	from twisted.internet.task import LoopingCall
22	22	from twisted.web.http import HTTPChannel
	23	from twisted.web.resource import Resource
23	24
24	25	from synapse.app.generic_worker import (
25	26	GenericWorkerReplicationHandler,

27	28	)
28	29	from synapse.http.server import JsonResource
29	30	from synapse.http.site import SynapseRequest, SynapseSite
30		from synapse.replication.http import ReplicationRestResource, streams
	31	from synapse.replication.http import ReplicationRestResource
31	32	from synapse.replication.tcp.handler import ReplicationCommandHandler
32	33	from synapse.replication.tcp.protocol import ClientReplicationStreamProtocol
33	34	from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory

53	54	if not hiredis:
54	55	skip = "Requires hiredis"
55	56
56		servlets = [
57		streams.register_servlets,
58		]
59
60	57	def prepare(self, reactor, clock, hs):
61	58	# build a replication server
62	59	server_factory = ReplicationStreamProtocolFactory(hs)

66	63	# Make a new HomeServer object for the worker
67	64	self.reactor.lookups["testserv"] = "1.2.3.4"
68	65	self.worker_hs = self.setup_test_homeserver(
69		http_client=None,
	66	federation_http_client=None,
70	67	homeserver_to_use=GenericWorkerServer,
71	68	config=self._get_worker_hs_config(),
72	69	reactor=self.reactor,

86	83
87	84	self._client_transport = None
88	85	self._server_transport = None
	86
	87	def create_resource_dict(self) -> Dict[str, Resource]:
	88	d = super().create_resource_dict()
	89	d["/_synapse/replication"] = ReplicationRestResource(self.hs)
	90	return d
89	91
90	92	def _get_worker_hs_config(self) -> dict:
91	93	config = self.default_config()

263	265	worker_app: Type of worker, e.g. `synapse.app.federation_sender`.
264	266	extra_config: Any extra config to use for this instances.
265	267	**kwargs: Options that get passed to `self.setup_test_homeserver`,
266		useful to e.g. pass some mocks for things like `http_client`
	268	useful to e.g. pass some mocks for things like `federation_http_client`
267	269
268	270	Returns:
269	271	The new worker HomeServer instance.

+117

-0

tests/replication/test_auth.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14	import logging
	15
	16	from synapse.rest.client.v2_alpha import register
	17
	18	from tests.replication._base import BaseMultiWorkerStreamTestCase
	19	from tests.server import FakeChannel, make_request
	20	from tests.unittest import override_config
	21
	22	logger = logging.getLogger(__name__)
	23
	24
	25	class WorkerAuthenticationTestCase(BaseMultiWorkerStreamTestCase):
	26	"""Test the authentication of HTTP calls between workers."""
	27
	28	servlets = [register.register_servlets]
	29
	30	def make_homeserver(self, reactor, clock):
	31	config = self.default_config()
	32	# This isn't a real configuration option but is used to provide the main
	33	# homeserver and worker homeserver different options.
	34	main_replication_secret = config.pop("main_replication_secret", None)
	35	if main_replication_secret:
	36	config["worker_replication_secret"] = main_replication_secret
	37	return self.setup_test_homeserver(config=config)
	38
	39	def _get_worker_hs_config(self) -> dict:
	40	config = self.default_config()
	41	config["worker_app"] = "synapse.app.client_reader"
	42	config["worker_replication_host"] = "testserv"
	43	config["worker_replication_http_port"] = "8765"
	44
	45	return config
	46
	47	def _test_register(self) -> FakeChannel:
	48	"""Run the actual test:
	49
	50	1. Create a worker homeserver.
	51	2. Start registration by providing a user/password.
	52	3. Complete registration by providing dummy auth (this hits the main synapse).
	53	4. Return the final request.
	54
	55	"""
	56	worker_hs = self.make_worker_hs("synapse.app.client_reader")
	57	site = self._hs_to_site[worker_hs]
	58
	59	channel_1 = make_request(
	60	self.reactor,
	61	site,
	62	"POST",
	63	"register",
	64	{"username": "user", "type": "m.login.password", "password": "bar"},
	65	)
	66	self.assertEqual(channel_1.code, 401)
	67
	68	# Grab the session
	69	session = channel_1.json_body["session"]
	70
	71	# also complete the dummy auth
	72	return make_request(
	73	self.reactor,
	74	site,
	75	"POST",
	76	"register",
	77	{"auth": {"session": session, "type": "m.login.dummy"}},
	78	)
	79
	80	def test_no_auth(self):
	81	"""With no authentication the request should finish.
	82	"""
	83	channel = self._test_register()
	84	self.assertEqual(channel.code, 200)
	85
	86	# We're given a registered user.
	87	self.assertEqual(channel.json_body["user_id"], "@user:test")
	88
	89	@override_config({"main_replication_secret": "my-secret"})
	90	def test_missing_auth(self):
	91	"""If the main process expects a secret that is not provided, an error results.
	92	"""
	93	channel = self._test_register()
	94	self.assertEqual(channel.code, 500)
	95
	96	@override_config(
	97	{
	98	"main_replication_secret": "my-secret",
	99	"worker_replication_secret": "wrong-secret",
	100	}
	101	)
	102	def test_unauthorized(self):
	103	"""If the main process receives the wrong secret, an error results.
	104	"""
	105	channel = self._test_register()
	106	self.assertEqual(channel.code, 500)
	107
	108	@override_config({"worker_replication_secret": "my-secret"})
	109	def test_authorized(self):
	110	"""The request should finish when the worker provides the authentication header.
	111	"""
	112	channel = self._test_register()
	113	self.assertEqual(channel.code, 200)
	114
	115	# We're given a registered user.
	116	self.assertEqual(channel.json_body["user_id"], "@user:test")

+14

-22

tests/replication/test_client_reader_shard.py less more

13	13	# limitations under the License.
14	14	import logging
15	15
16		from synapse.api.constants import LoginType
17		from synapse.http.site import SynapseRequest
18	16	from synapse.rest.client.v2_alpha import register
19	17
20	18	from tests.replication._base import BaseMultiWorkerStreamTestCase
21		from tests.rest.client.v2_alpha.test_auth import DummyRecaptchaChecker
22		from tests.server import FakeChannel, make_request
	19	from tests.server import make_request
23	20
24	21	logger = logging.getLogger(__name__)
25	22
26	23
27	24	class ClientReaderTestCase(BaseMultiWorkerStreamTestCase):
28		"""Base class for tests of the replication streams"""
	25	"""Test using one or more client readers for registration."""
29	26
30	27	servlets = [register.register_servlets]
31
32		def prepare(self, reactor, clock, hs):
33		self.recaptcha_checker = DummyRecaptchaChecker(hs)
34		auth_handler = hs.get_auth_handler()
35		auth_handler.checkers[LoginType.RECAPTCHA] = self.recaptcha_checker
36	28
37	29	def _get_worker_hs_config(self) -> dict:
38	30	config = self.default_config()

47	39	worker_hs = self.make_worker_hs("synapse.app.client_reader")
48	40	site = self._hs_to_site[worker_hs]
49	41
50		request_1, channel_1 = make_request(
	42	channel_1 = make_request(
51	43	self.reactor,
52	44	site,
53	45	"POST",
54	46	"register",
55	47	{"username": "user", "type": "m.login.password", "password": "bar"},
56		) # type: SynapseRequest, FakeChannel
57		self.assertEqual(request_1.code, 401)
	48	)
	49	self.assertEqual(channel_1.code, 401)
58	50
59	51	# Grab the session
60	52	session = channel_1.json_body["session"]
61	53
62	54	# also complete the dummy auth
63		request_2, channel_2 = make_request(
	55	channel_2 = make_request(
64	56	self.reactor,
65	57	site,
66	58	"POST",
67	59	"register",
68	60	{"auth": {"session": session, "type": "m.login.dummy"}},
69		) # type: SynapseRequest, FakeChannel
70		self.assertEqual(request_2.code, 200)
	61	)
	62	self.assertEqual(channel_2.code, 200)
71	63
72	64	# We're given a registered user.
73	65	self.assertEqual(channel_2.json_body["user_id"], "@user:test")

79	71	worker_hs_2 = self.make_worker_hs("synapse.app.client_reader")
80	72
81	73	site_1 = self._hs_to_site[worker_hs_1]
82		request_1, channel_1 = make_request(
	74	channel_1 = make_request(
83	75	self.reactor,
84	76	site_1,
85	77	"POST",
86	78	"register",
87	79	{"username": "user", "type": "m.login.password", "password": "bar"},
88		) # type: SynapseRequest, FakeChannel
89		self.assertEqual(request_1.code, 401)
	80	)
	81	self.assertEqual(channel_1.code, 401)
90	82
91	83	# Grab the session
92	84	session = channel_1.json_body["session"]
93	85
94	86	# also complete the dummy auth
95	87	site_2 = self._hs_to_site[worker_hs_2]
96		request_2, channel_2 = make_request(
	88	channel_2 = make_request(
97	89	self.reactor,
98	90	site_2,
99	91	"POST",
100	92	"register",
101	93	{"auth": {"session": session, "type": "m.login.dummy"}},
102		) # type: SynapseRequest, FakeChannel
103		self.assertEqual(request_2.code, 200)
	94	)
	95	self.assertEqual(channel_2.code, 200)
104	96
105	97	# We're given a registered user.
106	98	self.assertEqual(channel_2.json_body["user_id"], "@user:test")

+5

-5

tests/replication/test_federation_sender_shard.py less more

49	49	self.make_worker_hs(
50	50	"synapse.app.federation_sender",
51	51	{"send_federation": True},
52		http_client=mock_client,
	52	federation_http_client=mock_client,
53	53	)
54	54
55	55	user = self.register_user("user", "pass")

80	80	"worker_name": "sender1",
81	81	"federation_sender_instances": ["sender1", "sender2"],
82	82	},
83		http_client=mock_client1,
	83	federation_http_client=mock_client1,
84	84	)
85	85
86	86	mock_client2 = Mock(spec=["put_json"])

92	92	"worker_name": "sender2",
93	93	"federation_sender_instances": ["sender1", "sender2"],
94	94	},
95		http_client=mock_client2,
	95	federation_http_client=mock_client2,
96	96	)
97	97
98	98	user = self.register_user("user2", "pass")

143	143	"worker_name": "sender1",
144	144	"federation_sender_instances": ["sender1", "sender2"],
145	145	},
146		http_client=mock_client1,
	146	federation_http_client=mock_client1,
147	147	)
148	148
149	149	mock_client2 = Mock(spec=["put_json"])

155	155	"worker_name": "sender2",
156	156	"federation_sender_instances": ["sender1", "sender2"],
157	157	},
158		http_client=mock_client2,
	158	federation_http_client=mock_client2,
159	159	)
160	160
161	161	user = self.register_user("user3", "pass")

+2

-2

tests/replication/test_multi_media_repo.py less more

47	47	self.user_id = self.register_user("user", "pass")
48	48	self.access_token = self.login("user", "pass")
49	49
50		self.reactor.lookups["example.com"] = "127.0.0.2"
	50	self.reactor.lookups["example.com"] = "1.2.3.4"
51	51
52	52	def default_config(self):
53	53	conf = super().default_config()

67	67	the media which the caller should respond to.
68	68	"""
69	69	resource = hs.get_media_repository_resource().children[b"download"]
70		_, channel = make_request(
	70	channel = make_request(
71	71	self.reactor,
72	72	FakeSite(resource),
73	73	"GET",

+7

-7

tests/replication/test_pusher_shard.py less more

66	66	device_display_name="pushy push",
67	67	pushkey="a@example.com",
68	68	lang=None,
69		data={"url": "https://push.example.com/push"},
	69	data={"url": "https://push.example.com/_matrix/push/v1/notify"},
70	70	)
71	71	)
72	72

97	97	self.make_worker_hs(
98	98	"synapse.app.pusher",
99	99	{"start_pushers": True},
100		proxied_http_client=http_client_mock,
	100	proxied_blacklisted_http_client=http_client_mock,
101	101	)
102	102
103	103	event_id = self._create_pusher_and_send_msg("user")

108	108	http_client_mock.post_json_get_json.assert_called_once()
109	109	self.assertEqual(
110	110	http_client_mock.post_json_get_json.call_args[0][0],
111		"https://push.example.com/push",
	111	"https://push.example.com/_matrix/push/v1/notify",
112	112	)
113	113	self.assertEqual(
114	114	event_id,

132	132	"worker_name": "pusher1",
133	133	"pusher_instances": ["pusher1", "pusher2"],
134	134	},
135		proxied_http_client=http_client_mock1,
	135	proxied_blacklisted_http_client=http_client_mock1,
136	136	)
137	137
138	138	http_client_mock2 = Mock(spec_set=["post_json_get_json"])

147	147	"worker_name": "pusher2",
148	148	"pusher_instances": ["pusher1", "pusher2"],
149	149	},
150		proxied_http_client=http_client_mock2,
	150	proxied_blacklisted_http_client=http_client_mock2,
151	151	)
152	152
153	153	# We choose a user name that we know should go to pusher1.

160	160	http_client_mock2.post_json_get_json.assert_not_called()
161	161	self.assertEqual(
162	162	http_client_mock1.post_json_get_json.call_args[0][0],
163		"https://push.example.com/push",
	163	"https://push.example.com/_matrix/push/v1/notify",
164	164	)
165	165	self.assertEqual(
166	166	event_id,

182	182	http_client_mock2.post_json_get_json.assert_called_once()
183	183	self.assertEqual(
184	184	http_client_mock2.post_json_get_json.call_args[0][0],
185		"https://push.example.com/push",
	185	"https://push.example.com/_matrix/push/v1/notify",
186	186	)
187	187	self.assertEqual(
188	188	event_id,

+8

-8

tests/replication/test_sharded_event_persister.py less more

179	179	)
180	180
181	181	# Do an initial sync so that we're up to date.
182		request, channel = make_request(
	182	channel = make_request(
183	183	self.reactor, sync_hs_site, "GET", "/sync", access_token=access_token
184	184	)
185	185	next_batch = channel.json_body["next_batch"]

205	205
206	206	# Check that syncing still gets the new event, despite the gap in the
207	207	# stream IDs.
208		request, channel = make_request(
	208	channel = make_request(
209	209	self.reactor,
210	210	sync_hs_site,
211	211	"GET",

235	235	response = self.helper.send(room_id2, body="Hi!", tok=self.other_access_token)
236	236	first_event_in_room2 = response["event_id"]
237	237
238		request, channel = make_request(
	238	channel = make_request(
239	239	self.reactor,
240	240	sync_hs_site,
241	241	"GET",

260	260	self.helper.send(room_id1, body="Hi again!", tok=self.other_access_token)
261	261	self.helper.send(room_id2, body="Hi again!", tok=self.other_access_token)
262	262
263		request, channel = make_request(
	263	channel = make_request(
264	264	self.reactor,
265	265	sync_hs_site,
266	266	"GET",

278	278	# Paginating back in the first room should not produce any results, as
279	279	# no events have happened in it. This tests that we are correctly
280	280	# filtering results based on the vector clock portion.
281		request, channel = make_request(
	281	channel = make_request(
282	282	self.reactor,
283	283	sync_hs_site,
284	284	"GET",

291	291
292	292	# Paginating back on the second room should produce the first event
293	293	# again. This tests that pagination isn't completely broken.
294		request, channel = make_request(
	294	channel = make_request(
295	295	self.reactor,
296	296	sync_hs_site,
297	297	"GET",

306	306	)
307	307
308	308	# Paginating forwards should give the same results
309		request, channel = make_request(
	309	channel = make_request(
310	310	self.reactor,
311	311	sync_hs_site,
312	312	"GET",

317	317	)
318	318	self.assertListEqual([], channel.json_body["chunk"])
319	319
320		request, channel = make_request(
	320	channel = make_request(
321	321	self.reactor,
322	322	sync_hs_site,
323	323	"GET",

+17

-17

tests/rest/admin/test_admin.py less more

41	41	return resource
42	42
43	43	def test_version_string(self):
44		request, channel = self.make_request("GET", self.url, shorthand=False)
	44	channel = self.make_request("GET", self.url, shorthand=False)
45	45
46	46	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
47	47	self.assertEqual(

67	67
68	68	def test_delete_group(self):
69	69	# Create a new group
70		request, channel = self.make_request(
	70	channel = self.make_request(
71	71	"POST",
72	72	"/create_group".encode("ascii"),
73	73	access_token=self.admin_user_tok,

83	83	# Invite/join another user
84	84
85	85	url = "/groups/%s/admin/users/invite/%s" % (group_id, self.other_user)
86		request, channel = self.make_request(
	86	channel = self.make_request(
87	87	"PUT", url.encode("ascii"), access_token=self.admin_user_tok, content={}
88	88	)
89	89	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
90	90
91	91	url = "/groups/%s/self/accept_invite" % (group_id,)
92		request, channel = self.make_request(
	92	channel = self.make_request(
93	93	"PUT", url.encode("ascii"), access_token=self.other_user_token, content={}
94	94	)
95	95	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

100	100
101	101	# Now delete the group
102	102	url = "/_synapse/admin/v1/delete_group/" + group_id
103		request, channel = self.make_request(
	103	channel = self.make_request(
104	104	"POST",
105	105	url.encode("ascii"),
106	106	access_token=self.admin_user_tok,

122	122	"""
123	123
124	124	url = "/groups/%s/profile" % (group_id,)
125		request, channel = self.make_request(
	125	channel = self.make_request(
126	126	"GET", url.encode("ascii"), access_token=self.admin_user_tok
127	127	)
128	128

133	133	def _get_groups_user_is_in(self, access_token):
134	134	"""Returns the list of groups the user is in (given their access token)
135	135	"""
136		request, channel = self.make_request(
	136	channel = self.make_request(
137	137	"GET", "/joined_groups".encode("ascii"), access_token=access_token
138	138	)
139	139

209	209	}
210	210	config["media_storage_providers"] = [provider_config]
211	211
212		hs = self.setup_test_homeserver(config=config, http_client=client)
	212	hs = self.setup_test_homeserver(config=config, federation_http_client=client)
213	213
214	214	return hs
215	215
216	216	def _ensure_quarantined(self, admin_user_tok, server_and_media_id):
217	217	"""Ensure a piece of media is quarantined when trying to access it."""
218		request, channel = make_request(
	218	channel = make_request(
219	219	self.reactor,
220	220	FakeSite(self.download_resource),
221	221	"GET",

240	240
241	241	# Attempt quarantine media APIs as non-admin
242	242	url = "/_synapse/admin/v1/media/quarantine/example.org/abcde12345"
243		request, channel = self.make_request(
	243	channel = self.make_request(
244	244	"POST", url.encode("ascii"), access_token=non_admin_user_tok,
245	245	)
246	246

253	253
254	254	# And the roomID/userID endpoint
255	255	url = "/_synapse/admin/v1/room/!room%3Aexample.com/media/quarantine"
256		request, channel = self.make_request(
	256	channel = self.make_request(
257	257	"POST", url.encode("ascii"), access_token=non_admin_user_tok,
258	258	)
259	259

281	281	server_name, media_id = server_name_and_media_id.split("/")
282	282
283	283	# Attempt to access the media
284		request, channel = make_request(
	284	channel = make_request(
285	285	self.reactor,
286	286	FakeSite(self.download_resource),
287	287	"GET",

298	298	urllib.parse.quote(server_name),
299	299	urllib.parse.quote(media_id),
300	300	)
301		request, channel = self.make_request("POST", url, access_token=admin_user_tok,)
	301	channel = self.make_request("POST", url, access_token=admin_user_tok,)
302	302	self.pump(1.0)
303	303	self.assertEqual(200, int(channel.code), msg=channel.result["body"])
304	304

350	350	url = "/_synapse/admin/v1/room/%s/media/quarantine" % urllib.parse.quote(
351	351	room_id
352	352	)
353		request, channel = self.make_request("POST", url, access_token=admin_user_tok,)
	353	channel = self.make_request("POST", url, access_token=admin_user_tok,)
354	354	self.pump(1.0)
355	355	self.assertEqual(200, int(channel.code), msg=channel.result["body"])
356	356	self.assertEqual(

394	394	url = "/_synapse/admin/v1/user/%s/media/quarantine" % urllib.parse.quote(
395	395	non_admin_user
396	396	)
397		request, channel = self.make_request(
	397	channel = self.make_request(
398	398	"POST", url.encode("ascii"), access_token=admin_user_tok,
399	399	)
400	400	self.pump(1.0)

436	436	url = "/_synapse/admin/v1/user/%s/media/quarantine" % urllib.parse.quote(
437	437	non_admin_user
438	438	)
439		request, channel = self.make_request(
	439	channel = self.make_request(
440	440	"POST", url.encode("ascii"), access_token=admin_user_tok,
441	441	)
442	442	self.pump(1.0)

452	452	self._ensure_quarantined(admin_user_tok, server_and_media_id_1)
453	453
454	454	# Attempt to access each piece of media
455		request, channel = make_request(
	455	channel = make_request(
456	456	self.reactor,
457	457	FakeSite(self.download_resource),
458	458	"GET",

+50

-94

tests/rest/admin/test_device.py less more

49	49	"""
50	50	Try to get a device of an user without authentication.
51	51	"""
52		request, channel = self.make_request("GET", self.url, b"{}")
	52	channel = self.make_request("GET", self.url, b"{}")
53	53
54	54	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
55	55	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
56	56
57		request, channel = self.make_request("PUT", self.url, b"{}")
	57	channel = self.make_request("PUT", self.url, b"{}")
58	58
59	59	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
60	60	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
61	61
62		request, channel = self.make_request("DELETE", self.url, b"{}")
	62	channel = self.make_request("DELETE", self.url, b"{}")
63	63
64	64	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
65	65	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

68	68	"""
69	69	If the user is not a server admin, an error is returned.
70	70	"""
71		request, channel = self.make_request(
	71	channel = self.make_request(
72	72	"GET", self.url, access_token=self.other_user_token,
73	73	)
74	74
75	75	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
76	76	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
77	77
78		request, channel = self.make_request(
	78	channel = self.make_request(
79	79	"PUT", self.url, access_token=self.other_user_token,
80	80	)
81	81
82	82	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
83	83	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
84	84
85		request, channel = self.make_request(
	85	channel = self.make_request(
86	86	"DELETE", self.url, access_token=self.other_user_token,
87	87	)
88	88

98	98	% self.other_user_device_id
99	99	)
100	100
101		request, channel = self.make_request(
102		"GET", url, access_token=self.admin_user_tok,
103		)
104
105		self.assertEqual(404, channel.code, msg=channel.json_body)
106		self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
107
108		request, channel = self.make_request(
109		"PUT", url, access_token=self.admin_user_tok,
110		)
111
112		self.assertEqual(404, channel.code, msg=channel.json_body)
113		self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
114
115		request, channel = self.make_request(
116		"DELETE", url, access_token=self.admin_user_tok,
117		)
	101	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
	102
	103	self.assertEqual(404, channel.code, msg=channel.json_body)
	104	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
	105
	106	channel = self.make_request("PUT", url, access_token=self.admin_user_tok,)
	107
	108	self.assertEqual(404, channel.code, msg=channel.json_body)
	109	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
	110
	111	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
118	112
119	113	self.assertEqual(404, channel.code, msg=channel.json_body)
120	114	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

128	122	% self.other_user_device_id
129	123	)
130	124
131		request, channel = self.make_request(
132		"GET", url, access_token=self.admin_user_tok,
133		)
	125	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
134	126
135	127	self.assertEqual(400, channel.code, msg=channel.json_body)
136	128	self.assertEqual("Can only lookup local users", channel.json_body["error"])
137	129
138		request, channel = self.make_request(
139		"PUT", url, access_token=self.admin_user_tok,
140		)
	130	channel = self.make_request("PUT", url, access_token=self.admin_user_tok,)
141	131
142	132	self.assertEqual(400, channel.code, msg=channel.json_body)
143	133	self.assertEqual("Can only lookup local users", channel.json_body["error"])
144	134
145		request, channel = self.make_request(
146		"DELETE", url, access_token=self.admin_user_tok,
147		)
	135	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
148	136
149	137	self.assertEqual(400, channel.code, msg=channel.json_body)
150	138	self.assertEqual("Can only lookup local users", channel.json_body["error"])

157	145	self.other_user
158	146	)
159	147
160		request, channel = self.make_request(
161		"GET", url, access_token=self.admin_user_tok,
162		)
163
164		self.assertEqual(404, channel.code, msg=channel.json_body)
165		self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
166
167		request, channel = self.make_request(
168		"PUT", url, access_token=self.admin_user_tok,
169		)
170
171		self.assertEqual(200, channel.code, msg=channel.json_body)
172
173		request, channel = self.make_request(
174		"DELETE", url, access_token=self.admin_user_tok,
175		)
	148	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
	149
	150	self.assertEqual(404, channel.code, msg=channel.json_body)
	151	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])
	152
	153	channel = self.make_request("PUT", url, access_token=self.admin_user_tok,)
	154
	155	self.assertEqual(200, channel.code, msg=channel.json_body)
	156
	157	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
176	158
177	159	# Delete unknown device returns status 200
178	160	self.assertEqual(200, channel.code, msg=channel.json_body)

196	178	}
197	179
198	180	body = json.dumps(update)
199		request, channel = self.make_request(
	181	channel = self.make_request(
200	182	"PUT",
201	183	self.url,
202	184	access_token=self.admin_user_tok,

207	189	self.assertEqual(Codes.TOO_LARGE, channel.json_body["errcode"])
208	190
209	191	# Ensure the display name was not updated.
210		request, channel = self.make_request(
211		"GET", self.url, access_token=self.admin_user_tok,
212		)
	192	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
213	193
214	194	self.assertEqual(200, channel.code, msg=channel.json_body)
215	195	self.assertEqual("new display", channel.json_body["display_name"])

226	206	)
227	207	)
228	208
229		request, channel = self.make_request(
230		"PUT", self.url, access_token=self.admin_user_tok,
231		)
	209	channel = self.make_request("PUT", self.url, access_token=self.admin_user_tok,)
232	210
233	211	self.assertEqual(200, channel.code, msg=channel.json_body)
234	212
235	213	# Ensure the display name was not updated.
236		request, channel = self.make_request(
237		"GET", self.url, access_token=self.admin_user_tok,
238		)
	214	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
239	215
240	216	self.assertEqual(200, channel.code, msg=channel.json_body)
241	217	self.assertEqual("new display", channel.json_body["display_name"])

246	222	"""
247	223	# Set new display_name
248	224	body = json.dumps({"display_name": "new displayname"})
249		request, channel = self.make_request(
	225	channel = self.make_request(
250	226	"PUT",
251	227	self.url,
252	228	access_token=self.admin_user_tok,

256	232	self.assertEqual(200, channel.code, msg=channel.json_body)
257	233
258	234	# Check new display_name
259		request, channel = self.make_request(
260		"GET", self.url, access_token=self.admin_user_tok,
261		)
	235	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
262	236
263	237	self.assertEqual(200, channel.code, msg=channel.json_body)
264	238	self.assertEqual("new displayname", channel.json_body["display_name"])

267	241	"""
268	242	Tests that a normal lookup for a device is successfully
269	243	"""
270		request, channel = self.make_request(
271		"GET", self.url, access_token=self.admin_user_tok,
272		)
	244	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
273	245
274	246	self.assertEqual(200, channel.code, msg=channel.json_body)
275	247	self.assertEqual(self.other_user, channel.json_body["user_id"])

290	262	self.assertEqual(1, number_devices)
291	263
292	264	# Delete device
293		request, channel = self.make_request(
	265	channel = self.make_request(
294	266	"DELETE", self.url, access_token=self.admin_user_tok,
295	267	)
296	268

322	294	"""
323	295	Try to list devices of an user without authentication.
324	296	"""
325		request, channel = self.make_request("GET", self.url, b"{}")
	297	channel = self.make_request("GET", self.url, b"{}")
326	298
327	299	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
328	300	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

333	305	"""
334	306	other_user_token = self.login("user", "pass")
335	307
336		request, channel = self.make_request(
337		"GET", self.url, access_token=other_user_token,
338		)
	308	channel = self.make_request("GET", self.url, access_token=other_user_token,)
339	309
340	310	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
341	311	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

345	315	Tests that a lookup for a user that does not exist returns a 404
346	316	"""
347	317	url = "/_synapse/admin/v2/users/@unknown_person:test/devices"
348		request, channel = self.make_request(
349		"GET", url, access_token=self.admin_user_tok,
350		)
	318	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
351	319
352	320	self.assertEqual(404, channel.code, msg=channel.json_body)
353	321	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

358	326	"""
359	327	url = "/_synapse/admin/v2/users/@unknown_person:unknown_domain/devices"
360	328
361		request, channel = self.make_request(
362		"GET", url, access_token=self.admin_user_tok,
363		)
	329	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
364	330
365	331	self.assertEqual(400, channel.code, msg=channel.json_body)
366	332	self.assertEqual("Can only lookup local users", channel.json_body["error"])

372	338	"""
373	339
374	340	# Get devices
375		request, channel = self.make_request(
376		"GET", self.url, access_token=self.admin_user_tok,
377		)
	341	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
378	342
379	343	self.assertEqual(200, channel.code, msg=channel.json_body)
380	344	self.assertEqual(0, channel.json_body["total"])

390	354	self.login("user", "pass")
391	355
392	356	# Get devices
393		request, channel = self.make_request(
394		"GET", self.url, access_token=self.admin_user_tok,
395		)
	357	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
396	358
397	359	self.assertEqual(200, channel.code, msg=channel.json_body)
398	360	self.assertEqual(number_devices, channel.json_body["total"])

430	392	"""
431	393	Try to delete devices of an user without authentication.
432	394	"""
433		request, channel = self.make_request("POST", self.url, b"{}")
	395	channel = self.make_request("POST", self.url, b"{}")
434	396
435	397	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
436	398	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

441	403	"""
442	404	other_user_token = self.login("user", "pass")
443	405
444		request, channel = self.make_request(
445		"POST", self.url, access_token=other_user_token,
446		)
	406	channel = self.make_request("POST", self.url, access_token=other_user_token,)
447	407
448	408	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
449	409	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

453	413	Tests that a lookup for a user that does not exist returns a 404
454	414	"""
455	415	url = "/_synapse/admin/v2/users/@unknown_person:test/delete_devices"
456		request, channel = self.make_request(
457		"POST", url, access_token=self.admin_user_tok,
458		)
	416	channel = self.make_request("POST", url, access_token=self.admin_user_tok,)
459	417
460	418	self.assertEqual(404, channel.code, msg=channel.json_body)
461	419	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

466	424	"""
467	425	url = "/_synapse/admin/v2/users/@unknown_person:unknown_domain/delete_devices"
468	426
469		request, channel = self.make_request(
470		"POST", url, access_token=self.admin_user_tok,
471		)
	427	channel = self.make_request("POST", url, access_token=self.admin_user_tok,)
472	428
473	429	self.assertEqual(400, channel.code, msg=channel.json_body)
474	430	self.assertEqual("Can only lookup local users", channel.json_body["error"])

478	434	Tests that a remove of a device that does not exist returns 200.
479	435	"""
480	436	body = json.dumps({"devices": ["unknown_device1", "unknown_device2"]})
481		request, channel = self.make_request(
	437	channel = self.make_request(
482	438	"POST",
483	439	self.url,
484	440	access_token=self.admin_user_tok,

509	465
510	466	# Delete devices
511	467	body = json.dumps({"devices": device_ids})
512		request, channel = self.make_request(
	468	channel = self.make_request(
513	469	"POST",
514	470	self.url,
515	471	access_token=self.admin_user_tok,

+27

-35

tests/rest/admin/test_event_reports.py less more

73	73	"""
74	74	Try to get an event report without authentication.
75	75	"""
76		request, channel = self.make_request("GET", self.url, b"{}")
	76	channel = self.make_request("GET", self.url, b"{}")
77	77
78	78	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
79	79	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

83	83	If the user is not a server admin, an error 403 is returned.
84	84	"""
85	85
86		request, channel = self.make_request(
87		"GET", self.url, access_token=self.other_user_tok,
88		)
	86	channel = self.make_request("GET", self.url, access_token=self.other_user_tok,)
89	87
90	88	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
91	89	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

95	93	Testing list of reported events
96	94	"""
97	95
98		request, channel = self.make_request(
99		"GET", self.url, access_token=self.admin_user_tok,
100		)
	96	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
101	97
102	98	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
103	99	self.assertEqual(channel.json_body["total"], 20)

110	106	Testing list of reported events with limit
111	107	"""
112	108
113		request, channel = self.make_request(
	109	channel = self.make_request(
114	110	"GET", self.url + "?limit=5", access_token=self.admin_user_tok,
115	111	)
116	112

125	121	Testing list of reported events with a defined starting point (from)
126	122	"""
127	123
128		request, channel = self.make_request(
	124	channel = self.make_request(
129	125	"GET", self.url + "?from=5", access_token=self.admin_user_tok,
130	126	)
131	127

140	136	Testing list of reported events with a defined starting point and limit
141	137	"""
142	138
143		request, channel = self.make_request(
	139	channel = self.make_request(
144	140	"GET", self.url + "?from=5&limit=10", access_token=self.admin_user_tok,
145	141	)
146	142

155	151	Testing list of reported events with a filter of room
156	152	"""
157	153
158		request, channel = self.make_request(
	154	channel = self.make_request(
159	155	"GET",
160	156	self.url + "?room_id=%s" % self.room_id1,
161	157	access_token=self.admin_user_tok,

175	171	Testing list of reported events with a filter of user
176	172	"""
177	173
178		request, channel = self.make_request(
	174	channel = self.make_request(
179	175	"GET",
180	176	self.url + "?user_id=%s" % self.other_user,
181	177	access_token=self.admin_user_tok,

195	191	Testing list of reported events with a filter of user and room
196	192	"""
197	193
198		request, channel = self.make_request(
	194	channel = self.make_request(
199	195	"GET",
200	196	self.url + "?user_id=%s&room_id=%s" % (self.other_user, self.room_id1),
201	197	access_token=self.admin_user_tok,

217	213	"""
218	214
219	215	# fetch the most recent first, largest timestamp
220		request, channel = self.make_request(
	216	channel = self.make_request(
221	217	"GET", self.url + "?dir=b", access_token=self.admin_user_tok,
222	218	)
223	219

233	229	report += 1
234	230
235	231	# fetch the oldest first, smallest timestamp
236		request, channel = self.make_request(
	232	channel = self.make_request(
237	233	"GET", self.url + "?dir=f", access_token=self.admin_user_tok,
238	234	)
239	235

253	249	Testing that a invalid search order returns a 400
254	250	"""
255	251
256		request, channel = self.make_request(
	252	channel = self.make_request(
257	253	"GET", self.url + "?dir=bar", access_token=self.admin_user_tok,
258	254	)
259	255

266	262	Testing that a negative limit parameter returns a 400
267	263	"""
268	264
269		request, channel = self.make_request(
	265	channel = self.make_request(
270	266	"GET", self.url + "?limit=-5", access_token=self.admin_user_tok,
271	267	)
272	268

278	274	Testing that a negative from parameter returns a 400
279	275	"""
280	276
281		request, channel = self.make_request(
	277	channel = self.make_request(
282	278	"GET", self.url + "?from=-5", access_token=self.admin_user_tok,
283	279	)
284	280

292	288
293	289	# `next_token` does not appear
294	290	# Number of results is the number of entries
295		request, channel = self.make_request(
	291	channel = self.make_request(
296	292	"GET", self.url + "?limit=20", access_token=self.admin_user_tok,
297	293	)
298	294

303	299
304	300	# `next_token` does not appear
305	301	# Number of max results is larger than the number of entries
306		request, channel = self.make_request(
	302	channel = self.make_request(
307	303	"GET", self.url + "?limit=21", access_token=self.admin_user_tok,
308	304	)
309	305

314	310
315	311	# `next_token` does appear
316	312	# Number of max results is smaller than the number of entries
317		request, channel = self.make_request(
	313	channel = self.make_request(
318	314	"GET", self.url + "?limit=19", access_token=self.admin_user_tok,
319	315	)
320	316

326	322	# Check
327	323	# Set `from` to value of `next_token` for request remaining entries
328	324	# `next_token` does not appear
329		request, channel = self.make_request(
	325	channel = self.make_request(
330	326	"GET", self.url + "?from=19", access_token=self.admin_user_tok,
331	327	)
332	328

341	337	resp = self.helper.send(room_id, tok=user_tok)
342	338	event_id = resp["event_id"]
343	339
344		request, channel = self.make_request(
	340	channel = self.make_request(
345	341	"POST",
346	342	"rooms/%s/report/%s" % (room_id, event_id),
347	343	json.dumps({"score": -100, "reason": "this makes me sad"}),

398	394	"""
399	395	Try to get event report without authentication.
400	396	"""
401		request, channel = self.make_request("GET", self.url, b"{}")
	397	channel = self.make_request("GET", self.url, b"{}")
402	398
403	399	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
404	400	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

408	404	If the user is not a server admin, an error 403 is returned.
409	405	"""
410	406
411		request, channel = self.make_request(
412		"GET", self.url, access_token=self.other_user_tok,
413		)
	407	channel = self.make_request("GET", self.url, access_token=self.other_user_tok,)
414	408
415	409	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
416	410	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

420	414	Testing get a reported event
421	415	"""
422	416
423		request, channel = self.make_request(
424		"GET", self.url, access_token=self.admin_user_tok,
425		)
	417	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
426	418
427	419	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
428	420	self._check_fields(channel.json_body)

433	425	"""
434	426
435	427	# `report_id` is negative
436		request, channel = self.make_request(
	428	channel = self.make_request(
437	429	"GET",
438	430	"/_synapse/admin/v1/event_reports/-123",
439	431	access_token=self.admin_user_tok,

447	439	)
448	440
449	441	# `report_id` is a non-numerical string
450		request, channel = self.make_request(
	442	channel = self.make_request(
451	443	"GET",
452	444	"/_synapse/admin/v1/event_reports/abcdef",
453	445	access_token=self.admin_user_tok,

461	453	)
462	454
463	455	# `report_id` is undefined
464		request, channel = self.make_request(
	456	channel = self.make_request(
465	457	"GET",
466	458	"/_synapse/admin/v1/event_reports/",
467	459	access_token=self.admin_user_tok,

479	471	Testing that a not existing `report_id` returns a 404.
480	472	"""
481	473
482		request, channel = self.make_request(
	474	channel = self.make_request(
483	475	"GET",
484	476	"/_synapse/admin/v1/event_reports/123",
485	477	access_token=self.admin_user_tok,

495	487	resp = self.helper.send(room_id, tok=user_tok)
496	488	event_id = resp["event_id"]
497	489
498		request, channel = self.make_request(
	490	channel = self.make_request(
499	491	"POST",
500	492	"rooms/%s/report/%s" % (room_id, event_id),
501	493	json.dumps({"score": -100, "reason": "this makes me sad"}),

+26

-36

tests/rest/admin/test_media.py less more

49	49	"""
50	50	url = "/_synapse/admin/v1/media/%s/%s" % (self.server_name, "12345")
51	51
52		request, channel = self.make_request("DELETE", url, b"{}")
	52	channel = self.make_request("DELETE", url, b"{}")
53	53
54	54	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
55	55	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

63	63
64	64	url = "/_synapse/admin/v1/media/%s/%s" % (self.server_name, "12345")
65	65
66		request, channel = self.make_request(
67		"DELETE", url, access_token=self.other_user_token,
68		)
	66	channel = self.make_request("DELETE", url, access_token=self.other_user_token,)
69	67
70	68	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
71	69	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

76	74	"""
77	75	url = "/_synapse/admin/v1/media/%s/%s" % (self.server_name, "12345")
78	76
79		request, channel = self.make_request(
80		"DELETE", url, access_token=self.admin_user_tok,
81		)
	77	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
82	78
83	79	self.assertEqual(404, channel.code, msg=channel.json_body)
84	80	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

89	85	"""
90	86	url = "/_synapse/admin/v1/media/%s/%s" % ("unknown_domain", "12345")
91	87
92		request, channel = self.make_request(
93		"DELETE", url, access_token=self.admin_user_tok,
94		)
	88	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
95	89
96	90	self.assertEqual(400, channel.code, msg=channel.json_body)
97	91	self.assertEqual("Can only delete local media", channel.json_body["error"])

120	114	self.assertEqual(server_name, self.server_name)
121	115
122	116	# Attempt to access media
123		request, channel = make_request(
	117	channel = make_request(
124	118	self.reactor,
125	119	FakeSite(download_resource),
126	120	"GET",

145	139	url = "/_synapse/admin/v1/media/%s/%s" % (self.server_name, media_id)
146	140
147	141	# Delete media
148		request, channel = self.make_request(
149		"DELETE", url, access_token=self.admin_user_tok,
150		)
	142	channel = self.make_request("DELETE", url, access_token=self.admin_user_tok,)
151	143
152	144	self.assertEqual(200, channel.code, msg=channel.json_body)
153	145	self.assertEqual(1, channel.json_body["total"])

156	148	)
157	149
158	150	# Attempt to access media
159		request, channel = make_request(
	151	channel = make_request(
160	152	self.reactor,
161	153	FakeSite(download_resource),
162	154	"GET",

203	195	Try to delete media without authentication.
204	196	"""
205	197
206		request, channel = self.make_request("POST", self.url, b"{}")
	198	channel = self.make_request("POST", self.url, b"{}")
207	199
208	200	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
209	201	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

215	207	self.other_user = self.register_user("user", "pass")
216	208	self.other_user_token = self.login("user", "pass")
217	209
218		request, channel = self.make_request(
	210	channel = self.make_request(
219	211	"POST", self.url, access_token=self.other_user_token,
220	212	)
221	213

228	220	"""
229	221	url = "/_synapse/admin/v1/media/%s/delete" % "unknown_domain"
230	222
231		request, channel = self.make_request(
	223	channel = self.make_request(
232	224	"POST", url + "?before_ts=1234", access_token=self.admin_user_tok,
233	225	)
234	226

239	231	"""
240	232	If the parameter `before_ts` is missing, an error is returned.
241	233	"""
242		request, channel = self.make_request(
243		"POST", self.url, access_token=self.admin_user_tok,
244		)
	234	channel = self.make_request("POST", self.url, access_token=self.admin_user_tok,)
245	235
246	236	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
247	237	self.assertEqual(Codes.MISSING_PARAM, channel.json_body["errcode"])

253	243	"""
254	244	If parameters are invalid, an error is returned.
255	245	"""
256		request, channel = self.make_request(
	246	channel = self.make_request(
257	247	"POST", self.url + "?before_ts=-1234", access_token=self.admin_user_tok,
258	248	)
259	249

264	254	channel.json_body["error"],
265	255	)
266	256
267		request, channel = self.make_request(
	257	channel = self.make_request(
268	258	"POST",
269	259	self.url + "?before_ts=1234&size_gt=-1234",
270	260	access_token=self.admin_user_tok,

277	267	channel.json_body["error"],
278	268	)
279	269
280		request, channel = self.make_request(
	270	channel = self.make_request(
281	271	"POST",
282	272	self.url + "?before_ts=1234&keep_profiles=not_bool",
283	273	access_token=self.admin_user_tok,

307	297
308	298	# timestamp after upload/create
309	299	now_ms = self.clock.time_msec()
310		request, channel = self.make_request(
	300	channel = self.make_request(
311	301	"POST",
312	302	self.url + "?before_ts=" + str(now_ms),
313	303	access_token=self.admin_user_tok,

331	321
332	322	self._access_media(server_and_media_id)
333	323
334		request, channel = self.make_request(
	324	channel = self.make_request(
335	325	"POST",
336	326	self.url + "?before_ts=" + str(now_ms),
337	327	access_token=self.admin_user_tok,

343	333
344	334	# timestamp after upload
345	335	now_ms = self.clock.time_msec()
346		request, channel = self.make_request(
	336	channel = self.make_request(
347	337	"POST",
348	338	self.url + "?before_ts=" + str(now_ms),
349	339	access_token=self.admin_user_tok,

366	356	self._access_media(server_and_media_id)
367	357
368	358	now_ms = self.clock.time_msec()
369		request, channel = self.make_request(
	359	channel = self.make_request(
370	360	"POST",
371	361	self.url + "?before_ts=" + str(now_ms) + "&size_gt=67",
372	362	access_token=self.admin_user_tok,

377	367	self._access_media(server_and_media_id)
378	368
379	369	now_ms = self.clock.time_msec()
380		request, channel = self.make_request(
	370	channel = self.make_request(
381	371	"POST",
382	372	self.url + "?before_ts=" + str(now_ms) + "&size_gt=66",
383	373	access_token=self.admin_user_tok,

400	390	self._access_media(server_and_media_id)
401	391
402	392	# set media as avatar
403		request, channel = self.make_request(
	393	channel = self.make_request(
404	394	"PUT",
405	395	"/profile/%s/avatar_url" % (self.admin_user,),
406	396	content=json.dumps({"avatar_url": "mxc://%s" % (server_and_media_id,)}),

409	399	self.assertEqual(200, channel.code, msg=channel.json_body)
410	400
411	401	now_ms = self.clock.time_msec()
412		request, channel = self.make_request(
	402	channel = self.make_request(
413	403	"POST",
414	404	self.url + "?before_ts=" + str(now_ms) + "&keep_profiles=true",
415	405	access_token=self.admin_user_tok,

420	410	self._access_media(server_and_media_id)
421	411
422	412	now_ms = self.clock.time_msec()
423		request, channel = self.make_request(
	413	channel = self.make_request(
424	414	"POST",
425	415	self.url + "?before_ts=" + str(now_ms) + "&keep_profiles=false",
426	416	access_token=self.admin_user_tok,

444	434
445	435	# set media as room avatar
446	436	room_id = self.helper.create_room_as(self.admin_user, tok=self.admin_user_tok)
447		request, channel = self.make_request(
	437	channel = self.make_request(
448	438	"PUT",
449	439	"/rooms/%s/state/m.room.avatar" % (room_id,),
450	440	content=json.dumps({"url": "mxc://%s" % (server_and_media_id,)}),

453	443	self.assertEqual(200, channel.code, msg=channel.json_body)
454	444
455	445	now_ms = self.clock.time_msec()
456		request, channel = self.make_request(
	446	channel = self.make_request(
457	447	"POST",
458	448	self.url + "?before_ts=" + str(now_ms) + "&keep_profiles=true",
459	449	access_token=self.admin_user_tok,

464	454	self._access_media(server_and_media_id)
465	455
466	456	now_ms = self.clock.time_msec()
467		request, channel = self.make_request(
	457	channel = self.make_request(
468	458	"POST",
469	459	self.url + "?before_ts=" + str(now_ms) + "&keep_profiles=false",
470	460	access_token=self.admin_user_tok,

511	501	media_id = server_and_media_id.split("/")[1]
512	502	local_path = self.filepaths.local_media_filepath(media_id)
513	503
514		request, channel = make_request(
	504	channel = make_request(
515	505	self.reactor,
516	506	FakeSite(download_resource),
517	507	"GET",

+225

-47

tests/rest/admin/test_room.py less more

19	19	from mock import Mock
20	20
21	21	import synapse.rest.admin
	22	from synapse.api.constants import EventTypes, Membership
22	23	from synapse.api.errors import Codes
23	24	from synapse.rest.client.v1 import directory, events, login, room
24	25

78	79
79	80	# Test that the admin can still send shutdown
80	81	url = "/_synapse/admin/v1/shutdown_room/" + room_id
81		request, channel = self.make_request(
	82	channel = self.make_request(
82	83	"POST",
83	84	url.encode("ascii"),
84	85	json.dumps({"new_room_user_id": self.admin_user}),

102	103
103	104	# Enable world readable
104	105	url = "rooms/%s/state/m.room.history_visibility" % (room_id,)
105		request, channel = self.make_request(
	106	channel = self.make_request(
106	107	"PUT",
107	108	url.encode("ascii"),
108	109	json.dumps({"history_visibility": "world_readable"}),

112	113
113	114	# Test that the admin can still send shutdown
114	115	url = "/_synapse/admin/v1/shutdown_room/" + room_id
115		request, channel = self.make_request(
	116	channel = self.make_request(
116	117	"POST",
117	118	url.encode("ascii"),
118	119	json.dumps({"new_room_user_id": self.admin_user}),

129	130	"""
130	131
131	132	url = "rooms/%s/initialSync" % (room_id,)
132		request, channel = self.make_request(
	133	channel = self.make_request(
133	134	"GET", url.encode("ascii"), access_token=self.admin_user_tok
134	135	)
135	136	self.assertEqual(

137	138	)
138	139
139	140	url = "events?timeout=0&room_id=" + room_id
140		request, channel = self.make_request(
	141	channel = self.make_request(
141	142	"GET", url.encode("ascii"), access_token=self.admin_user_tok
142	143	)
143	144	self.assertEqual(

183	184	If the user is not a server admin, an error 403 is returned.
184	185	"""
185	186
186		request, channel = self.make_request(
	187	channel = self.make_request(
187	188	"POST", self.url, json.dumps({}), access_token=self.other_user_tok,
188	189	)
189	190

196	197	"""
197	198	url = "/_synapse/admin/v1/rooms/!unknown:test/delete"
198	199
199		request, channel = self.make_request(
	200	channel = self.make_request(
200	201	"POST", url, json.dumps({}), access_token=self.admin_user_tok,
201	202	)
202	203

209	210	"""
210	211	url = "/_synapse/admin/v1/rooms/invalidroom/delete"
211	212
212		request, channel = self.make_request(
	213	channel = self.make_request(
213	214	"POST", url, json.dumps({}), access_token=self.admin_user_tok,
214	215	)
215	216

224	225	"""
225	226	body = json.dumps({"new_room_user_id": "@unknown:test"})
226	227
227		request, channel = self.make_request(
	228	channel = self.make_request(
228	229	"POST",
229	230	self.url,
230	231	content=body.encode(encoding="utf_8"),

243	244	"""
244	245	body = json.dumps({"new_room_user_id": "@not:exist.bla"})
245	246
246		request, channel = self.make_request(
	247	channel = self.make_request(
247	248	"POST",
248	249	self.url,
249	250	content=body.encode(encoding="utf_8"),

261	262	"""
262	263	body = json.dumps({"block": "NotBool"})
263	264
264		request, channel = self.make_request(
	265	channel = self.make_request(
265	266	"POST",
266	267	self.url,
267	268	content=body.encode(encoding="utf_8"),

277	278	"""
278	279	body = json.dumps({"purge": "NotBool"})
279	280
280		request, channel = self.make_request(
	281	channel = self.make_request(
281	282	"POST",
282	283	self.url,
283	284	content=body.encode(encoding="utf_8"),

303	304
304	305	body = json.dumps({"block": True, "purge": True})
305	306
306		request, channel = self.make_request(
	307	channel = self.make_request(
307	308	"POST",
308	309	self.url.encode("ascii"),
309	310	content=body.encode(encoding="utf_8"),

336	337
337	338	body = json.dumps({"block": False, "purge": True})
338	339
339		request, channel = self.make_request(
	340	channel = self.make_request(
340	341	"POST",
341	342	self.url.encode("ascii"),
342	343	content=body.encode(encoding="utf_8"),

370	371
371	372	body = json.dumps({"block": False, "purge": False})
372	373
373		request, channel = self.make_request(
	374	channel = self.make_request(
374	375	"POST",
375	376	self.url.encode("ascii"),
376	377	content=body.encode(encoding="utf_8"),

417	418
418	419	# Test that the admin can still send shutdown
419	420	url = "/_synapse/admin/v1/rooms/%s/delete" % self.room_id
420		request, channel = self.make_request(
	421	channel = self.make_request(
421	422	"POST",
422	423	url.encode("ascii"),
423	424	json.dumps({"new_room_user_id": self.admin_user}),

447	448
448	449	# Enable world readable
449	450	url = "rooms/%s/state/m.room.history_visibility" % (self.room_id,)
450		request, channel = self.make_request(
	451	channel = self.make_request(
451	452	"PUT",
452	453	url.encode("ascii"),
453	454	json.dumps({"history_visibility": "world_readable"}),

464	465
465	466	# Test that the admin can still send shutdown
466	467	url = "/_synapse/admin/v1/rooms/%s/delete" % self.room_id
467		request, channel = self.make_request(
	468	channel = self.make_request(
468	469	"POST",
469	470	url.encode("ascii"),
470	471	json.dumps({"new_room_user_id": self.admin_user}),

529	530	"""
530	531
531	532	url = "rooms/%s/initialSync" % (room_id,)
532		request, channel = self.make_request(
	533	channel = self.make_request(
533	534	"GET", url.encode("ascii"), access_token=self.admin_user_tok
534	535	)
535	536	self.assertEqual(

537	538	)
538	539
539	540	url = "events?timeout=0&room_id=" + room_id
540		request, channel = self.make_request(
	541	channel = self.make_request(
541	542	"GET", url.encode("ascii"), access_token=self.admin_user_tok
542	543	)
543	544	self.assertEqual(

568	569	self.helper.leave(room_id, user=self.admin_user, tok=self.admin_user_tok)
569	570
570	571	url = "/_synapse/admin/v1/purge_room"
571		request, channel = self.make_request(
	572	channel = self.make_request(
572	573	"POST",
573	574	url.encode("ascii"),
574	575	{"room_id": room_id},

622	623
623	624	# Request the list of rooms
624	625	url = "/_synapse/admin/v1/rooms"
625		request, channel = self.make_request(
	626	channel = self.make_request(
626	627	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
627	628	)
628	629

703	704	limit,
704	705	"name",
705	706	)
706		request, channel = self.make_request(
	707	channel = self.make_request(
707	708	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
708	709	)
709	710	self.assertEqual(

743	744	self.assertEqual(room_ids, returned_room_ids)
744	745
745	746	url = "/_synapse/admin/v1/rooms?from=%d&limit=%d" % (start, limit)
746		request, channel = self.make_request(
	747	channel = self.make_request(
747	748	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
748	749	)
749	750	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

763	764
764	765	# Create a new alias to this room
765	766	url = "/_matrix/client/r0/directory/room/%s" % (urllib.parse.quote(test_alias),)
766		request, channel = self.make_request(
	767	channel = self.make_request(
767	768	"PUT",
768	769	url.encode("ascii"),
769	770	{"room_id": room_id},

793	794
794	795	# Request the list of rooms
795	796	url = "/_synapse/admin/v1/rooms"
796		request, channel = self.make_request(
	797	channel = self.make_request(
797	798	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
798	799	)
799	800	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

834	835	url = "/_matrix/client/r0/directory/room/%s" % (
835	836	urllib.parse.quote(test_alias),
836	837	)
837		request, channel = self.make_request(
	838	channel = self.make_request(
838	839	"PUT",
839	840	url.encode("ascii"),
840	841	{"room_id": room_id},

874	875	url = "/_synapse/admin/v1/rooms?order_by=%s" % (order_type,)
875	876	if reverse:
876	877	url += "&dir=b"
877		request, channel = self.make_request(
	878	channel = self.make_request(
878	879	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
879	880	)
880	881	self.assertEqual(200, channel.code, msg=channel.json_body)

1010	1011	expected_http_code: The expected http code for the request
1011	1012	"""
1012	1013	url = "/_synapse/admin/v1/rooms?search_term=%s" % (search_term,)
1013		request, channel = self.make_request(
	1014	channel = self.make_request(
1014	1015	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
1015	1016	)
1016	1017	self.assertEqual(expected_http_code, channel.code, msg=channel.json_body)

1048	1049
1049	1050	_search_test(room_id_2, "else")
1050	1051	_search_test(room_id_2, "se")
	1052
	1053	# Test case insensitive
	1054	_search_test(room_id_1, "SOMETHING")
	1055	_search_test(room_id_1, "THING")
	1056
	1057	_search_test(room_id_2, "ELSE")
	1058	_search_test(room_id_2, "SE")
1051	1059
1052	1060	_search_test(None, "foo")
1053	1061	_search_test(None, "bar")

1071	1079	)
1072	1080
1073	1081	url = "/_synapse/admin/v1/rooms/%s" % (room_id_1,)
1074		request, channel = self.make_request(
	1082	channel = self.make_request(
1075	1083	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
1076	1084	)
1077	1085	self.assertEqual(200, channel.code, msg=channel.json_body)

1083	1091	self.assertIn("canonical_alias", channel.json_body)
1084	1092	self.assertIn("joined_members", channel.json_body)
1085	1093	self.assertIn("joined_local_members", channel.json_body)
	1094	self.assertIn("joined_local_devices", channel.json_body)
1086	1095	self.assertIn("version", channel.json_body)
1087	1096	self.assertIn("creator", channel.json_body)
1088	1097	self.assertIn("encryption", channel.json_body)

1095	1104
1096	1105	self.assertEqual(room_id_1, channel.json_body["room_id"])
1097	1106
	1107	def test_single_room_devices(self):
	1108	"""Test that `joined_local_devices` can be requested correctly"""
	1109	room_id_1 = self.helper.create_room_as(self.admin_user, tok=self.admin_user_tok)
	1110
	1111	url = "/_synapse/admin/v1/rooms/%s" % (room_id_1,)
	1112	channel = self.make_request(
	1113	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
	1114	)
	1115	self.assertEqual(200, channel.code, msg=channel.json_body)
	1116	self.assertEqual(1, channel.json_body["joined_local_devices"])
	1117
	1118	# Have another user join the room
	1119	user_1 = self.register_user("foo", "pass")
	1120	user_tok_1 = self.login("foo", "pass")
	1121	self.helper.join(room_id_1, user_1, tok=user_tok_1)
	1122
	1123	url = "/_synapse/admin/v1/rooms/%s" % (room_id_1,)
	1124	channel = self.make_request(
	1125	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
	1126	)
	1127	self.assertEqual(200, channel.code, msg=channel.json_body)
	1128	self.assertEqual(2, channel.json_body["joined_local_devices"])
	1129
	1130	# leave room
	1131	self.helper.leave(room_id_1, self.admin_user, tok=self.admin_user_tok)
	1132	self.helper.leave(room_id_1, user_1, tok=user_tok_1)
	1133	url = "/_synapse/admin/v1/rooms/%s" % (room_id_1,)
	1134	channel = self.make_request(
	1135	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
	1136	)
	1137	self.assertEqual(200, channel.code, msg=channel.json_body)
	1138	self.assertEqual(0, channel.json_body["joined_local_devices"])
	1139
1098	1140	def test_room_members(self):
1099	1141	"""Test that room members can be requested correctly"""
1100	1142	# Create two test rooms

1118	1160	self.helper.join(room_id_2, user_3, tok=user_tok_3)
1119	1161
1120	1162	url = "/_synapse/admin/v1/rooms/%s/members" % (room_id_1,)
1121		request, channel = self.make_request(
	1163	channel = self.make_request(
1122	1164	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
1123	1165	)
1124	1166	self.assertEqual(200, channel.code, msg=channel.json_body)

1129	1171	self.assertEqual(channel.json_body["total"], 3)
1130	1172
1131	1173	url = "/_synapse/admin/v1/rooms/%s/members" % (room_id_2,)
1132		request, channel = self.make_request(
	1174	channel = self.make_request(
1133	1175	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
1134	1176	)
1135	1177	self.assertEqual(200, channel.code, msg=channel.json_body)

1169	1211	"""
1170	1212	body = json.dumps({"user_id": self.second_user_id})
1171	1213
1172		request, channel = self.make_request(
	1214	channel = self.make_request(
1173	1215	"POST",
1174	1216	self.url,
1175	1217	content=body.encode(encoding="utf_8"),

1185	1227	"""
1186	1228	body = json.dumps({"unknown_parameter": "@unknown:test"})
1187	1229
1188		request, channel = self.make_request(
	1230	channel = self.make_request(
1189	1231	"POST",
1190	1232	self.url,
1191	1233	content=body.encode(encoding="utf_8"),

1201	1243	"""
1202	1244	body = json.dumps({"user_id": "@unknown:test"})
1203	1245
1204		request, channel = self.make_request(
	1246	channel = self.make_request(
1205	1247	"POST",
1206	1248	self.url,
1207	1249	content=body.encode(encoding="utf_8"),

1217	1259	"""
1218	1260	body = json.dumps({"user_id": "@not:exist.bla"})
1219	1261
1220		request, channel = self.make_request(
	1262	channel = self.make_request(
1221	1263	"POST",
1222	1264	self.url,
1223	1265	content=body.encode(encoding="utf_8"),

1237	1279	body = json.dumps({"user_id": self.second_user_id})
1238	1280	url = "/_synapse/admin/v1/join/!unknown:test"
1239	1281
1240		request, channel = self.make_request(
	1282	channel = self.make_request(
1241	1283	"POST",
1242	1284	url,
1243	1285	content=body.encode(encoding="utf_8"),

1254	1296	body = json.dumps({"user_id": self.second_user_id})
1255	1297	url = "/_synapse/admin/v1/join/invalidroom"
1256	1298
1257		request, channel = self.make_request(
	1299	channel = self.make_request(
1258	1300	"POST",
1259	1301	url,
1260	1302	content=body.encode(encoding="utf_8"),

1273	1315	"""
1274	1316	body = json.dumps({"user_id": self.second_user_id})
1275	1317
1276		request, channel = self.make_request(
	1318	channel = self.make_request(
1277	1319	"POST",
1278	1320	self.url,
1279	1321	content=body.encode(encoding="utf_8"),

1285	1327
1286	1328	# Validate if user is a member of the room
1287	1329
1288		request, channel = self.make_request(
	1330	channel = self.make_request(
1289	1331	"GET", "/_matrix/client/r0/joined_rooms", access_token=self.second_tok,
1290	1332	)
1291	1333	self.assertEquals(200, int(channel.result["code"]), msg=channel.result["body"])

1302	1344	url = "/_synapse/admin/v1/join/{}".format(private_room_id)
1303	1345	body = json.dumps({"user_id": self.second_user_id})
1304	1346
1305		request, channel = self.make_request(
	1347	channel = self.make_request(
1306	1348	"POST",
1307	1349	url,
1308	1350	content=body.encode(encoding="utf_8"),

1332	1374
1333	1375	# Validate if server admin is a member of the room
1334	1376
1335		request, channel = self.make_request(
	1377	channel = self.make_request(
1336	1378	"GET", "/_matrix/client/r0/joined_rooms", access_token=self.admin_user_tok,
1337	1379	)
1338	1380	self.assertEquals(200, int(channel.result["code"]), msg=channel.result["body"])

1343	1385	url = "/_synapse/admin/v1/join/{}".format(private_room_id)
1344	1386	body = json.dumps({"user_id": self.second_user_id})
1345	1387
1346		request, channel = self.make_request(
	1388	channel = self.make_request(
1347	1389	"POST",
1348	1390	url,
1349	1391	content=body.encode(encoding="utf_8"),

1354	1396
1355	1397	# Validate if user is a member of the room
1356	1398
1357		request, channel = self.make_request(
	1399	channel = self.make_request(
1358	1400	"GET", "/_matrix/client/r0/joined_rooms", access_token=self.second_tok,
1359	1401	)
1360	1402	self.assertEquals(200, int(channel.result["code"]), msg=channel.result["body"])

1371	1413	url = "/_synapse/admin/v1/join/{}".format(private_room_id)
1372	1414	body = json.dumps({"user_id": self.second_user_id})
1373	1415
1374		request, channel = self.make_request(
	1416	channel = self.make_request(
1375	1417	"POST",
1376	1418	url,
1377	1419	content=body.encode(encoding="utf_8"),

1383	1425
1384	1426	# Validate if user is a member of the room
1385	1427
1386		request, channel = self.make_request(
	1428	channel = self.make_request(
1387	1429	"GET", "/_matrix/client/r0/joined_rooms", access_token=self.second_tok,
1388	1430	)
1389	1431	self.assertEquals(200, int(channel.result["code"]), msg=channel.result["body"])
1390	1432	self.assertEqual(private_room_id, channel.json_body["joined_rooms"][0])
	1433
	1434
	1435	class MakeRoomAdminTestCase(unittest.HomeserverTestCase):
	1436	servlets = [
	1437	synapse.rest.admin.register_servlets,
	1438	room.register_servlets,
	1439	login.register_servlets,
	1440	]
	1441
	1442	def prepare(self, reactor, clock, homeserver):
	1443	self.admin_user = self.register_user("admin", "pass", admin=True)
	1444	self.admin_user_tok = self.login("admin", "pass")
	1445
	1446	self.creator = self.register_user("creator", "test")
	1447	self.creator_tok = self.login("creator", "test")
	1448
	1449	self.second_user_id = self.register_user("second", "test")
	1450	self.second_tok = self.login("second", "test")
	1451
	1452	self.public_room_id = self.helper.create_room_as(
	1453	self.creator, tok=self.creator_tok, is_public=True
	1454	)
	1455	self.url = "/_synapse/admin/v1/rooms/{}/make_room_admin".format(
	1456	self.public_room_id
	1457	)
	1458
	1459	def test_public_room(self):
	1460	"""Test that getting admin in a public room works.
	1461	"""
	1462	room_id = self.helper.create_room_as(
	1463	self.creator, tok=self.creator_tok, is_public=True
	1464	)
	1465
	1466	channel = self.make_request(
	1467	"POST",
	1468	"/_synapse/admin/v1/rooms/{}/make_room_admin".format(room_id),
	1469	content={},
	1470	access_token=self.admin_user_tok,
	1471	)
	1472
	1473	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	1474
	1475	# Now we test that we can join the room and ban a user.
	1476	self.helper.join(room_id, self.admin_user, tok=self.admin_user_tok)
	1477	self.helper.change_membership(
	1478	room_id,
	1479	self.admin_user,
	1480	"@test:test",
	1481	Membership.BAN,
	1482	tok=self.admin_user_tok,
	1483	)
	1484
	1485	def test_private_room(self):
	1486	"""Test that getting admin in a private room works and we get invited.
	1487	"""
	1488	room_id = self.helper.create_room_as(
	1489	self.creator, tok=self.creator_tok, is_public=False,
	1490	)
	1491
	1492	channel = self.make_request(
	1493	"POST",
	1494	"/_synapse/admin/v1/rooms/{}/make_room_admin".format(room_id),
	1495	content={},
	1496	access_token=self.admin_user_tok,
	1497	)
	1498
	1499	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	1500
	1501	# Now we test that we can join the room (we should have received an
	1502	# invite) and can ban a user.
	1503	self.helper.join(room_id, self.admin_user, tok=self.admin_user_tok)
	1504	self.helper.change_membership(
	1505	room_id,
	1506	self.admin_user,
	1507	"@test:test",
	1508	Membership.BAN,
	1509	tok=self.admin_user_tok,
	1510	)
	1511
	1512	def test_other_user(self):
	1513	"""Test that giving admin in a public room works to a non-admin user works.
	1514	"""
	1515	room_id = self.helper.create_room_as(
	1516	self.creator, tok=self.creator_tok, is_public=True
	1517	)
	1518
	1519	channel = self.make_request(
	1520	"POST",
	1521	"/_synapse/admin/v1/rooms/{}/make_room_admin".format(room_id),
	1522	content={"user_id": self.second_user_id},
	1523	access_token=self.admin_user_tok,
	1524	)
	1525
	1526	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	1527
	1528	# Now we test that we can join the room and ban a user.
	1529	self.helper.join(room_id, self.second_user_id, tok=self.second_tok)
	1530	self.helper.change_membership(
	1531	room_id,
	1532	self.second_user_id,
	1533	"@test:test",
	1534	Membership.BAN,
	1535	tok=self.second_tok,
	1536	)
	1537
	1538	def test_not_enough_power(self):
	1539	"""Test that we get a sensible error if there are no local room admins.
	1540	"""
	1541	room_id = self.helper.create_room_as(
	1542	self.creator, tok=self.creator_tok, is_public=True
	1543	)
	1544
	1545	# The creator drops admin rights in the room.
	1546	pl = self.helper.get_state(
	1547	room_id, EventTypes.PowerLevels, tok=self.creator_tok
	1548	)
	1549	pl["users"][self.creator] = 0
	1550	self.helper.send_state(
	1551	room_id, EventTypes.PowerLevels, body=pl, tok=self.creator_tok
	1552	)
	1553
	1554	channel = self.make_request(
	1555	"POST",
	1556	"/_synapse/admin/v1/rooms/{}/make_room_admin".format(room_id),
	1557	content={},
	1558	access_token=self.admin_user_tok,
	1559	)
	1560
	1561	# We expect this to fail with a 400 as there are no room admins.
	1562	#
	1563	# (Note we assert the error message to ensure that it's not denied for
	1564	# some other reason)
	1565	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
	1566	self.assertEqual(
	1567	channel.json_body["error"],
	1568	"No local admin user in room with power to update power levels.",
	1569	)
1391	1570
1392	1571
1393	1572	PURGE_TABLES = [

1418	1597	"event_push_summary",
1419	1598	"pusher_throttle",
1420	1599	"group_summary_rooms",
1421		"local_invites",
1422	1600	"room_account_data",
1423	1601	"room_tags",
1424	1602	# "state_groups", # Current impl leaves orphaned state groups around.

+27

-33

tests/rest/admin/test_statistics.py less more

45	45	"""
46	46	Try to list users without authentication.
47	47	"""
48		request, channel = self.make_request("GET", self.url, b"{}")
	48	channel = self.make_request("GET", self.url, b"{}")
49	49
50	50	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
51	51	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

54	54	"""
55	55	If the user is not a server admin, an error 403 is returned.
56	56	"""
57		request, channel = self.make_request(
	57	channel = self.make_request(
58	58	"GET", self.url, json.dumps({}), access_token=self.other_user_tok,
59	59	)
60	60

66	66	If parameters are invalid, an error is returned.
67	67	"""
68	68	# unkown order_by
69		request, channel = self.make_request(
	69	channel = self.make_request(
70	70	"GET", self.url + "?order_by=bar", access_token=self.admin_user_tok,
71	71	)
72	72

74	74	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
75	75
76	76	# negative from
77		request, channel = self.make_request(
	77	channel = self.make_request(
78	78	"GET", self.url + "?from=-5", access_token=self.admin_user_tok,
79	79	)
80	80

82	82	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
83	83
84	84	# negative limit
85		request, channel = self.make_request(
	85	channel = self.make_request(
86	86	"GET", self.url + "?limit=-5", access_token=self.admin_user_tok,
87	87	)
88	88

90	90	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
91	91
92	92	# negative from_ts
93		request, channel = self.make_request(
	93	channel = self.make_request(
94	94	"GET", self.url + "?from_ts=-1234", access_token=self.admin_user_tok,
95	95	)
96	96

98	98	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
99	99
100	100	# negative until_ts
101		request, channel = self.make_request(
	101	channel = self.make_request(
102	102	"GET", self.url + "?until_ts=-1234", access_token=self.admin_user_tok,
103	103	)
104	104

106	106	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
107	107
108	108	# until_ts smaller from_ts
109		request, channel = self.make_request(
	109	channel = self.make_request(
110	110	"GET",
111	111	self.url + "?from_ts=10&until_ts=5",
112	112	access_token=self.admin_user_tok,

116	116	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
117	117
118	118	# empty search term
119		request, channel = self.make_request(
	119	channel = self.make_request(
120	120	"GET", self.url + "?search_term=", access_token=self.admin_user_tok,
121	121	)
122	122

124	124	self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
125	125
126	126	# invalid search order
127		request, channel = self.make_request(
	127	channel = self.make_request(
128	128	"GET", self.url + "?dir=bar", access_token=self.admin_user_tok,
129	129	)
130	130

137	137	"""
138	138	self._create_users_with_media(10, 2)
139	139
140		request, channel = self.make_request(
	140	channel = self.make_request(
141	141	"GET", self.url + "?limit=5", access_token=self.admin_user_tok,
142	142	)
143	143

153	153	"""
154	154	self._create_users_with_media(20, 2)
155	155
156		request, channel = self.make_request(
	156	channel = self.make_request(
157	157	"GET", self.url + "?from=5", access_token=self.admin_user_tok,
158	158	)
159	159

169	169	"""
170	170	self._create_users_with_media(20, 2)
171	171
172		request, channel = self.make_request(
	172	channel = self.make_request(
173	173	"GET", self.url + "?from=5&limit=10", access_token=self.admin_user_tok,
174	174	)
175	175

189	189
190	190	# `next_token` does not appear
191	191	# Number of results is the number of entries
192		request, channel = self.make_request(
	192	channel = self.make_request(
193	193	"GET", self.url + "?limit=20", access_token=self.admin_user_tok,
194	194	)
195	195

200	200
201	201	# `next_token` does not appear
202	202	# Number of max results is larger than the number of entries
203		request, channel = self.make_request(
	203	channel = self.make_request(
204	204	"GET", self.url + "?limit=21", access_token=self.admin_user_tok,
205	205	)
206	206

211	211
212	212	# `next_token` does appear
213	213	# Number of max results is smaller than the number of entries
214		request, channel = self.make_request(
	214	channel = self.make_request(
215	215	"GET", self.url + "?limit=19", access_token=self.admin_user_tok,
216	216	)
217	217

222	222
223	223	# Set `from` to value of `next_token` for request remaining entries
224	224	# Check `next_token` does not appear
225		request, channel = self.make_request(
	225	channel = self.make_request(
226	226	"GET", self.url + "?from=19", access_token=self.admin_user_tok,
227	227	)
228	228

237	237	if users have no media created
238	238	"""
239	239
240		request, channel = self.make_request(
241		"GET", self.url, access_token=self.admin_user_tok,
242		)
	240	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
243	241
244	242	self.assertEqual(200, channel.code, msg=channel.json_body)
245	243	self.assertEqual(0, channel.json_body["total"])

315	313	ts1 = self.clock.time_msec()
316	314
317	315	# list all media when filter is not set
318		request, channel = self.make_request(
319		"GET", self.url, access_token=self.admin_user_tok,
320		)
	316	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
321	317	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
322	318	self.assertEqual(channel.json_body["users"][0]["media_count"], 3)
323	319
324	320	# filter media starting at `ts1` after creating first media
325	321	# result is 0
326		request, channel = self.make_request(
	322	channel = self.make_request(
327	323	"GET", self.url + "?from_ts=%s" % (ts1,), access_token=self.admin_user_tok,
328	324	)
329	325	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

336	332	self._create_media(self.other_user_tok, 3)
337	333
338	334	# filter media between `ts1` and `ts2`
339		request, channel = self.make_request(
	335	channel = self.make_request(
340	336	"GET",
341	337	self.url + "?from_ts=%s&until_ts=%s" % (ts1, ts2),
342	338	access_token=self.admin_user_tok,

345	341	self.assertEqual(channel.json_body["users"][0]["media_count"], 3)
346	342
347	343	# filter media until `ts2` and earlier
348		request, channel = self.make_request(
	344	channel = self.make_request(
349	345	"GET", self.url + "?until_ts=%s" % (ts2,), access_token=self.admin_user_tok,
350	346	)
351	347	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

355	351	self._create_users_with_media(20, 1)
356	352
357	353	# check without filter get all users
358		request, channel = self.make_request(
359		"GET", self.url, access_token=self.admin_user_tok,
360		)
	354	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
361	355	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
362	356	self.assertEqual(channel.json_body["total"], 20)
363	357
364	358	# filter user 1 and 10-19 by `user_id`
365		request, channel = self.make_request(
	359	channel = self.make_request(
366	360	"GET",
367	361	self.url + "?search_term=foo_user_1",
368	362	access_token=self.admin_user_tok,

371	365	self.assertEqual(channel.json_body["total"], 11)
372	366
373	367	# filter on this user in `displayname`
374		request, channel = self.make_request(
	368	channel = self.make_request(
375	369	"GET",
376	370	self.url + "?search_term=bar_user_10",
377	371	access_token=self.admin_user_tok,

381	375	self.assertEqual(channel.json_body["total"], 1)
382	376
383	377	# filter and get empty result
384		request, channel = self.make_request(
	378	channel = self.make_request(
385	379	"GET", self.url + "?search_term=foobar", access_token=self.admin_user_tok,
386	380	)
387	381	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

446	440	url = self.url + "?order_by=%s" % (order_type,)
447	441	if dir is not None and dir in ("b", "f"):
448	442	url += "&dir=%s" % (dir,)
449		request, channel = self.make_request(
	443	channel = self.make_request(
450	444	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
451	445	)
452	446	self.assertEqual(200, channel.code, msg=channel.json_body)

+274

-200

tests/rest/admin/test_user.py less more

17	17	import json
18	18	import urllib.parse
19	19	from binascii import unhexlify
	20	from typing import Optional
20	21
21	22	from mock import Mock
22	23

69	70	"""
70	71	self.hs.config.registration_shared_secret = None
71	72
72		request, channel = self.make_request("POST", self.url, b"{}")
	73	channel = self.make_request("POST", self.url, b"{}")
73	74
74	75	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
75	76	self.assertEqual(

86	87
87	88	self.hs.get_secrets = Mock(return_value=secrets)
88	89
89		request, channel = self.make_request("GET", self.url)
	90	channel = self.make_request("GET", self.url)
90	91
91	92	self.assertEqual(channel.json_body, {"nonce": "abcd"})
92	93

95	96	Calling GET on the endpoint will return a randomised nonce, which will
96	97	only last for SALT_TIMEOUT (60s).
97	98	"""
98		request, channel = self.make_request("GET", self.url)
	99	channel = self.make_request("GET", self.url)
99	100	nonce = channel.json_body["nonce"]
100	101
101	102	# 59 seconds
102	103	self.reactor.advance(59)
103	104
104	105	body = json.dumps({"nonce": nonce})
105		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	106	channel = self.make_request("POST", self.url, body.encode("utf8"))
106	107
107	108	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
108	109	self.assertEqual("username must be specified", channel.json_body["error"])

110	111	# 61 seconds
111	112	self.reactor.advance(2)
112	113
113		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	114	channel = self.make_request("POST", self.url, body.encode("utf8"))
114	115
115	116	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
116	117	self.assertEqual("unrecognised nonce", channel.json_body["error"])

119	120	"""
120	121	Only the provided nonce can be used, as it's checked in the MAC.
121	122	"""
122		request, channel = self.make_request("GET", self.url)
	123	channel = self.make_request("GET", self.url)
123	124	nonce = channel.json_body["nonce"]
124	125
125	126	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

135	136	"mac": want_mac,
136	137	}
137	138	)
138		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	139	channel = self.make_request("POST", self.url, body.encode("utf8"))
139	140
140	141	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
141	142	self.assertEqual("HMAC incorrect", channel.json_body["error"])

145	146	When the correct nonce is provided, and the right key is provided, the
146	147	user is registered.
147	148	"""
148		request, channel = self.make_request("GET", self.url)
	149	channel = self.make_request("GET", self.url)
149	150	nonce = channel.json_body["nonce"]
150	151
151	152	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

164	165	"mac": want_mac,
165	166	}
166	167	)
167		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	168	channel = self.make_request("POST", self.url, body.encode("utf8"))
168	169
169	170	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
170	171	self.assertEqual("@bob:test", channel.json_body["user_id"])

173	174	"""
174	175	A valid unrecognised nonce.
175	176	"""
176		request, channel = self.make_request("GET", self.url)
	177	channel = self.make_request("GET", self.url)
177	178	nonce = channel.json_body["nonce"]
178	179
179	180	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

189	190	"mac": want_mac,
190	191	}
191	192	)
192		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	193	channel = self.make_request("POST", self.url, body.encode("utf8"))
193	194
194	195	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
195	196	self.assertEqual("@bob:test", channel.json_body["user_id"])
196	197
197	198	# Now, try and reuse it
198		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	199	channel = self.make_request("POST", self.url, body.encode("utf8"))
199	200
200	201	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
201	202	self.assertEqual("unrecognised nonce", channel.json_body["error"])

208	209	"""
209	210
210	211	def nonce():
211		request, channel = self.make_request("GET", self.url)
	212	channel = self.make_request("GET", self.url)
212	213	return channel.json_body["nonce"]
213	214
214	215	#

217	218
218	219	# Must be present
219	220	body = json.dumps({})
220		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	221	channel = self.make_request("POST", self.url, body.encode("utf8"))
221	222
222	223	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
223	224	self.assertEqual("nonce must be specified", channel.json_body["error"])

228	229
229	230	# Must be present
230	231	body = json.dumps({"nonce": nonce()})
231		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	232	channel = self.make_request("POST", self.url, body.encode("utf8"))
232	233
233	234	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
234	235	self.assertEqual("username must be specified", channel.json_body["error"])
235	236
236	237	# Must be a string
237	238	body = json.dumps({"nonce": nonce(), "username": 1234})
238		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	239	channel = self.make_request("POST", self.url, body.encode("utf8"))
239	240
240	241	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
241	242	self.assertEqual("Invalid username", channel.json_body["error"])
242	243
243	244	# Must not have null bytes
244	245	body = json.dumps({"nonce": nonce(), "username": "abcd\u0000"})
245		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	246	channel = self.make_request("POST", self.url, body.encode("utf8"))
246	247
247	248	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
248	249	self.assertEqual("Invalid username", channel.json_body["error"])
249	250
250	251	# Must not have null bytes
251	252	body = json.dumps({"nonce": nonce(), "username": "a" * 1000})
252		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	253	channel = self.make_request("POST", self.url, body.encode("utf8"))
253	254
254	255	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
255	256	self.assertEqual("Invalid username", channel.json_body["error"])

260	261
261	262	# Must be present
262	263	body = json.dumps({"nonce": nonce(), "username": "a"})
263		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	264	channel = self.make_request("POST", self.url, body.encode("utf8"))
264	265
265	266	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
266	267	self.assertEqual("password must be specified", channel.json_body["error"])
267	268
268	269	# Must be a string
269	270	body = json.dumps({"nonce": nonce(), "username": "a", "password": 1234})
270		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	271	channel = self.make_request("POST", self.url, body.encode("utf8"))
271	272
272	273	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
273	274	self.assertEqual("Invalid password", channel.json_body["error"])
274	275
275	276	# Must not have null bytes
276	277	body = json.dumps({"nonce": nonce(), "username": "a", "password": "abcd\u0000"})
277		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	278	channel = self.make_request("POST", self.url, body.encode("utf8"))
278	279
279	280	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
280	281	self.assertEqual("Invalid password", channel.json_body["error"])
281	282
282	283	# Super long
283	284	body = json.dumps({"nonce": nonce(), "username": "a", "password": "A" * 1000})
284		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	285	channel = self.make_request("POST", self.url, body.encode("utf8"))
285	286
286	287	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
287	288	self.assertEqual("Invalid password", channel.json_body["error"])

299	300	"user_type": "invalid",
300	301	}
301	302	)
302		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	303	channel = self.make_request("POST", self.url, body.encode("utf8"))
303	304
304	305	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
305	306	self.assertEqual("Invalid user type", channel.json_body["error"])

310	311	"""
311	312
312	313	# set no displayname
313		request, channel = self.make_request("GET", self.url)
	314	channel = self.make_request("GET", self.url)
314	315	nonce = channel.json_body["nonce"]
315	316
316	317	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

320	321	body = json.dumps(
321	322	{"nonce": nonce, "username": "bob1", "password": "abc123", "mac": want_mac}
322	323	)
323		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	324	channel = self.make_request("POST", self.url, body.encode("utf8"))
324	325
325	326	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
326	327	self.assertEqual("@bob1:test", channel.json_body["user_id"])
327	328
328		request, channel = self.make_request("GET", "/profile/@bob1:test/displayname")
	329	channel = self.make_request("GET", "/profile/@bob1:test/displayname")
329	330	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
330	331	self.assertEqual("bob1", channel.json_body["displayname"])
331	332
332	333	# displayname is None
333		request, channel = self.make_request("GET", self.url)
	334	channel = self.make_request("GET", self.url)
334	335	nonce = channel.json_body["nonce"]
335	336
336	337	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

346	347	"mac": want_mac,
347	348	}
348	349	)
349		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	350	channel = self.make_request("POST", self.url, body.encode("utf8"))
350	351
351	352	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
352	353	self.assertEqual("@bob2:test", channel.json_body["user_id"])
353	354
354		request, channel = self.make_request("GET", "/profile/@bob2:test/displayname")
	355	channel = self.make_request("GET", "/profile/@bob2:test/displayname")
355	356	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
356	357	self.assertEqual("bob2", channel.json_body["displayname"])
357	358
358	359	# displayname is empty
359		request, channel = self.make_request("GET", self.url)
	360	channel = self.make_request("GET", self.url)
360	361	nonce = channel.json_body["nonce"]
361	362
362	363	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

372	373	"mac": want_mac,
373	374	}
374	375	)
375		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	376	channel = self.make_request("POST", self.url, body.encode("utf8"))
376	377
377	378	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
378	379	self.assertEqual("@bob3:test", channel.json_body["user_id"])
379	380
380		request, channel = self.make_request("GET", "/profile/@bob3:test/displayname")
	381	channel = self.make_request("GET", "/profile/@bob3:test/displayname")
381	382	self.assertEqual(404, int(channel.result["code"]), msg=channel.result["body"])
382	383
383	384	# set displayname
384		request, channel = self.make_request("GET", self.url)
	385	channel = self.make_request("GET", self.url)
385	386	nonce = channel.json_body["nonce"]
386	387
387	388	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

397	398	"mac": want_mac,
398	399	}
399	400	)
400		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	401	channel = self.make_request("POST", self.url, body.encode("utf8"))
401	402
402	403	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
403	404	self.assertEqual("@bob4:test", channel.json_body["user_id"])
404	405
405		request, channel = self.make_request("GET", "/profile/@bob4:test/displayname")
	406	channel = self.make_request("GET", "/profile/@bob4:test/displayname")
406	407	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
407	408	self.assertEqual("Bob's Name", channel.json_body["displayname"])
408	409

428	429	)
429	430
430	431	# Register new user with admin API
431		request, channel = self.make_request("GET", self.url)
	432	channel = self.make_request("GET", self.url)
432	433	nonce = channel.json_body["nonce"]
433	434
434	435	want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1)

447	448	"mac": want_mac,
448	449	}
449	450	)
450		request, channel = self.make_request("POST", self.url, body.encode("utf8"))
	451	channel = self.make_request("POST", self.url, body.encode("utf8"))
451	452
452	453	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
453	454	self.assertEqual("@bob:test", channel.json_body["user_id"])

465	466	self.admin_user = self.register_user("admin", "pass", admin=True)
466	467	self.admin_user_tok = self.login("admin", "pass")
467	468
468		self.register_user("user1", "pass1", admin=False)
469		self.register_user("user2", "pass2", admin=False)
	469	self.user1 = self.register_user(
	470	"user1", "pass1", admin=False, displayname="Name 1"
	471	)
	472	self.user2 = self.register_user(
	473	"user2", "pass2", admin=False, displayname="Name 2"
	474	)
470	475
471	476	def test_no_auth(self):
472	477	"""
473	478	Try to list users without authentication.
474	479	"""
475		request, channel = self.make_request("GET", self.url, b"{}")
	480	channel = self.make_request("GET", self.url, b"{}")
476	481
477	482	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
478		self.assertEqual("M_MISSING_TOKEN", channel.json_body["errcode"])
	483	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
	484
	485	def test_requester_is_no_admin(self):
	486	"""
	487	If the user is not a server admin, an error is returned.
	488	"""
	489	other_user_token = self.login("user1", "pass1")
	490
	491	channel = self.make_request("GET", self.url, access_token=other_user_token)
	492
	493	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
	494	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
479	495
480	496	def test_all_users(self):
481	497	"""
482	498	List all users, including deactivated users.
483	499	"""
484		request, channel = self.make_request(
	500	channel = self.make_request(
485	501	"GET",
486	502	self.url + "?deactivated=true",
487	503	b"{}",

491	507	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
492	508	self.assertEqual(3, len(channel.json_body["users"]))
493	509	self.assertEqual(3, channel.json_body["total"])
	510
	511	# Check that all fields are available
	512	for u in channel.json_body["users"]:
	513	self.assertIn("name", u)
	514	self.assertIn("is_guest", u)
	515	self.assertIn("admin", u)
	516	self.assertIn("user_type", u)
	517	self.assertIn("deactivated", u)
	518	self.assertIn("displayname", u)
	519	self.assertIn("avatar_url", u)
	520
	521	def test_search_term(self):
	522	"""Test that searching for a users works correctly"""
	523
	524	def _search_test(
	525	expected_user_id: Optional[str],
	526	search_term: str,
	527	search_field: Optional[str] = "name",
	528	expected_http_code: Optional[int] = 200,
	529	):
	530	"""Search for a user and check that the returned user's id is a match
	531
	532	Args:
	533	expected_user_id: The user_id expected to be returned by the API. Set
	534	to None to expect zero results for the search
	535	search_term: The term to search for user names with
	536	search_field: Field which is to request: `name` or `user_id`
	537	expected_http_code: The expected http code for the request
	538	"""
	539	url = self.url + "?%s=%s" % (search_field, search_term,)
	540	channel = self.make_request(
	541	"GET", url.encode("ascii"), access_token=self.admin_user_tok,
	542	)
	543	self.assertEqual(expected_http_code, channel.code, msg=channel.json_body)
	544
	545	if expected_http_code != 200:
	546	return
	547
	548	# Check that users were returned
	549	self.assertTrue("users" in channel.json_body)
	550	users = channel.json_body["users"]
	551
	552	# Check that the expected number of users were returned
	553	expected_user_count = 1 if expected_user_id else 0
	554	self.assertEqual(len(users), expected_user_count)
	555	self.assertEqual(channel.json_body["total"], expected_user_count)
	556
	557	if expected_user_id:
	558	# Check that the first returned user id is correct
	559	u = users[0]
	560	self.assertEqual(expected_user_id, u["name"])
	561
	562	# Perform search tests
	563	_search_test(self.user1, "er1")
	564	_search_test(self.user1, "me 1")
	565
	566	_search_test(self.user2, "er2")
	567	_search_test(self.user2, "me 2")
	568
	569	_search_test(self.user1, "er1", "user_id")
	570	_search_test(self.user2, "er2", "user_id")
	571
	572	# Test case insensitive
	573	_search_test(self.user1, "ER1")
	574	_search_test(self.user1, "NAME 1")
	575
	576	_search_test(self.user2, "ER2")
	577	_search_test(self.user2, "NAME 2")
	578
	579	_search_test(self.user1, "ER1", "user_id")
	580	_search_test(self.user2, "ER2", "user_id")
	581
	582	_search_test(None, "foo")
	583	_search_test(None, "bar")
	584
	585	_search_test(None, "foo", "user_id")
	586	_search_test(None, "bar", "user_id")
494	587
495	588
496	589	class UserRestTestCase(unittest.HomeserverTestCase):

507	600	self.admin_user = self.register_user("admin", "pass", admin=True)
508	601	self.admin_user_tok = self.login("admin", "pass")
509	602
510		self.other_user = self.register_user("user", "pass")
	603	self.other_user = self.register_user("user", "pass", displayname="User")
511	604	self.other_user_token = self.login("user", "pass")
512	605	self.url_other_user = "/_synapse/admin/v2/users/%s" % urllib.parse.quote(
513	606	self.other_user

519	612	"""
520	613	url = "/_synapse/admin/v2/users/@bob:test"
521	614
522		request, channel = self.make_request(
523		"GET", url, access_token=self.other_user_token,
524		)
	615	channel = self.make_request("GET", url, access_token=self.other_user_token,)
525	616
526	617	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
527	618	self.assertEqual("You are not a server admin", channel.json_body["error"])
528	619
529		request, channel = self.make_request(
	620	channel = self.make_request(
530	621	"PUT", url, access_token=self.other_user_token, content=b"{}",
531	622	)
532	623

538	629	Tests that a lookup for a user that does not exist returns a 404
539	630	"""
540	631
541		request, channel = self.make_request(
	632	channel = self.make_request(
542	633	"GET",
543	634	"/_synapse/admin/v2/users/@unknown_person:test",
544	635	access_token=self.admin_user_tok,

560	651	"admin": True,
561	652	"displayname": "Bob's name",
562	653	"threepids": [{"medium": "email", "address": "bob@bob.bob"}],
563		"avatar_url": None,
	654	"avatar_url": "mxc://fibble/wibble",
564	655	}
565	656	)
566	657
567		request, channel = self.make_request(
	658	channel = self.make_request(
568	659	"PUT",
569	660	url,
570	661	access_token=self.admin_user_tok,

577	668	self.assertEqual("email", channel.json_body["threepids"][0]["medium"])
578	669	self.assertEqual("bob@bob.bob", channel.json_body["threepids"][0]["address"])
579	670	self.assertEqual(True, channel.json_body["admin"])
	671	self.assertEqual("mxc://fibble/wibble", channel.json_body["avatar_url"])
580	672
581	673	# Get user
582		request, channel = self.make_request(
583		"GET", url, access_token=self.admin_user_tok,
584		)
	674	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
585	675
586	676	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
587	677	self.assertEqual("@bob:test", channel.json_body["name"])

591	681	self.assertEqual(True, channel.json_body["admin"])
592	682	self.assertEqual(False, channel.json_body["is_guest"])
593	683	self.assertEqual(False, channel.json_body["deactivated"])
	684	self.assertEqual("mxc://fibble/wibble", channel.json_body["avatar_url"])
594	685
595	686	def test_create_user(self):
596	687	"""

605	696	"admin": False,
606	697	"displayname": "Bob's name",
607	698	"threepids": [{"medium": "email", "address": "bob@bob.bob"}],
	699	"avatar_url": "mxc://fibble/wibble",
608	700	}
609	701	)
610	702
611		request, channel = self.make_request(
	703	channel = self.make_request(
612	704	"PUT",
613	705	url,
614	706	access_token=self.admin_user_tok,

621	713	self.assertEqual("email", channel.json_body["threepids"][0]["medium"])
622	714	self.assertEqual("bob@bob.bob", channel.json_body["threepids"][0]["address"])
623	715	self.assertEqual(False, channel.json_body["admin"])
	716	self.assertEqual("mxc://fibble/wibble", channel.json_body["avatar_url"])
624	717
625	718	# Get user
626		request, channel = self.make_request(
627		"GET", url, access_token=self.admin_user_tok,
628		)
	719	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
629	720
630	721	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
631	722	self.assertEqual("@bob:test", channel.json_body["name"])

635	726	self.assertEqual(False, channel.json_body["admin"])
636	727	self.assertEqual(False, channel.json_body["is_guest"])
637	728	self.assertEqual(False, channel.json_body["deactivated"])
	729	self.assertEqual("mxc://fibble/wibble", channel.json_body["avatar_url"])
638	730
639	731	@override_config(
640	732	{"limit_usage_by_mau": True, "max_mau_value": 2, "mau_trial_days": 0}

650	742
651	743	# Sync to set admin user to active
652	744	# before limit of monthly active users is reached
653		request, channel = self.make_request(
654		"GET", "/sync", access_token=self.admin_user_tok
655		)
	745	channel = self.make_request("GET", "/sync", access_token=self.admin_user_tok)
656	746
657	747	if channel.code != 200:
658	748	raise HttpResponseException(

675	765	# Create user
676	766	body = json.dumps({"password": "abc123", "admin": False})
677	767
678		request, channel = self.make_request(
	768	channel = self.make_request(
679	769	"PUT",
680	770	url,
681	771	access_token=self.admin_user_tok,

714	804	# Create user
715	805	body = json.dumps({"password": "abc123", "admin": False})
716	806
717		request, channel = self.make_request(
	807	channel = self.make_request(
718	808	"PUT",
719	809	url,
720	810	access_token=self.admin_user_tok,

751	841	}
752	842	)
753	843
754		request, channel = self.make_request(
	844	channel = self.make_request(
755	845	"PUT",
756	846	url,
757	847	access_token=self.admin_user_tok,

768	858	)
769	859	pushers = list(pushers)
770	860	self.assertEqual(len(pushers), 1)
771		self.assertEqual("@bob:test", pushers[0]["user_name"])
	861	self.assertEqual("@bob:test", pushers[0].user_name)
772	862
773	863	@override_config(
774	864	{

795	885	}
796	886	)
797	887
798		request, channel = self.make_request(
	888	channel = self.make_request(
799	889	"PUT",
800	890	url,
801	891	access_token=self.admin_user_tok,

821	911	# Change password
822	912	body = json.dumps({"password": "hahaha"})
823	913
824		request, channel = self.make_request(
	914	channel = self.make_request(
825	915	"PUT",
826	916	self.url_other_user,
827	917	access_token=self.admin_user_tok,

838	928	# Modify user
839	929	body = json.dumps({"displayname": "foobar"})
840	930
841		request, channel = self.make_request(
	931	channel = self.make_request(
842	932	"PUT",
843	933	self.url_other_user,
844	934	access_token=self.admin_user_tok,

850	940	self.assertEqual("foobar", channel.json_body["displayname"])
851	941
852	942	# Get user
853		request, channel = self.make_request(
	943	channel = self.make_request(
854	944	"GET", self.url_other_user, access_token=self.admin_user_tok,
855	945	)
856	946

868	958	{"threepids": [{"medium": "email", "address": "bob3@bob.bob"}]}
869	959	)
870	960
871		request, channel = self.make_request(
	961	channel = self.make_request(
872	962	"PUT",
873	963	self.url_other_user,
874	964	access_token=self.admin_user_tok,

881	971	self.assertEqual("bob3@bob.bob", channel.json_body["threepids"][0]["address"])
882	972
883	973	# Get user
884		request, channel = self.make_request(
	974	channel = self.make_request(
885	975	"GET", self.url_other_user, access_token=self.admin_user_tok,
886	976	)
887	977

898	988	# Deactivate user
899	989	body = json.dumps({"deactivated": True})
900	990
901		request, channel = self.make_request(
	991	channel = self.make_request(
902	992	"PUT",
903	993	self.url_other_user,
904	994	access_token=self.admin_user_tok,

911	1001	# the user is deactivated, the threepid will be deleted
912	1002
913	1003	# Get user
914		request, channel = self.make_request(
	1004	channel = self.make_request(
915	1005	"GET", self.url_other_user, access_token=self.admin_user_tok,
916	1006	)
917	1007

919	1009	self.assertEqual("@user:test", channel.json_body["name"])
920	1010	self.assertEqual(True, channel.json_body["deactivated"])
921	1011
	1012	@override_config({"user_directory": {"enabled": True, "search_all_users": True}})
	1013	def test_change_name_deactivate_user_user_directory(self):
	1014	"""
	1015	Test change profile information of a deactivated user and
	1016	check that it does not appear in user directory
	1017	"""
	1018
	1019	# is in user directory
	1020	profile = self.get_success(self.store.get_user_in_directory(self.other_user))
	1021	self.assertTrue(profile["display_name"] == "User")
	1022
	1023	# Deactivate user
	1024	body = json.dumps({"deactivated": True})
	1025
	1026	channel = self.make_request(
	1027	"PUT",
	1028	self.url_other_user,
	1029	access_token=self.admin_user_tok,
	1030	content=body.encode(encoding="utf_8"),
	1031	)
	1032
	1033	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	1034	self.assertEqual("@user:test", channel.json_body["name"])
	1035	self.assertEqual(True, channel.json_body["deactivated"])
	1036
	1037	# is not in user directory
	1038	profile = self.get_success(self.store.get_user_in_directory(self.other_user))
	1039	self.assertTrue(profile is None)
	1040
	1041	# Set new displayname user
	1042	body = json.dumps({"displayname": "Foobar"})
	1043
	1044	channel = self.make_request(
	1045	"PUT",
	1046	self.url_other_user,
	1047	access_token=self.admin_user_tok,
	1048	content=body.encode(encoding="utf_8"),
	1049	)
	1050
	1051	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
	1052	self.assertEqual("@user:test", channel.json_body["name"])
	1053	self.assertEqual(True, channel.json_body["deactivated"])
	1054	self.assertEqual("Foobar", channel.json_body["displayname"])
	1055
	1056	# is not in user directory
	1057	profile = self.get_success(self.store.get_user_in_directory(self.other_user))
	1058	self.assertTrue(profile is None)
	1059
922	1060	def test_reactivate_user(self):
923	1061	"""
924	1062	Test reactivating another user.
925	1063	"""
926	1064
927	1065	# Deactivate the user.
928		request, channel = self.make_request(
	1066	channel = self.make_request(
929	1067	"PUT",
930	1068	self.url_other_user,
931	1069	access_token=self.admin_user_tok,

938	1076	self._is_erased("@user:test", True)
939	1077
940	1078	# Attempt to reactivate the user (without a password).
941		request, channel = self.make_request(
	1079	channel = self.make_request(
942	1080	"PUT",
943	1081	self.url_other_user,
944	1082	access_token=self.admin_user_tok,

947	1085	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
948	1086
949	1087	# Reactivate the user.
950		request, channel = self.make_request(
	1088	channel = self.make_request(
951	1089	"PUT",
952	1090	self.url_other_user,
953	1091	access_token=self.admin_user_tok,

958	1096	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
959	1097
960	1098	# Get user
961		request, channel = self.make_request(
	1099	channel = self.make_request(
962	1100	"GET", self.url_other_user, access_token=self.admin_user_tok,
963	1101	)
964	1102

975	1113	# Set a user as an admin
976	1114	body = json.dumps({"admin": True})
977	1115
978		request, channel = self.make_request(
	1116	channel = self.make_request(
979	1117	"PUT",
980	1118	self.url_other_user,
981	1119	access_token=self.admin_user_tok,

987	1125	self.assertEqual(True, channel.json_body["admin"])
988	1126
989	1127	# Get user
990		request, channel = self.make_request(
	1128	channel = self.make_request(
991	1129	"GET", self.url_other_user, access_token=self.admin_user_tok,
992	1130	)
993	1131

1005	1143	# Create user
1006	1144	body = json.dumps({"password": "abc123"})
1007	1145
1008		request, channel = self.make_request(
	1146	channel = self.make_request(
1009	1147	"PUT",
1010	1148	url,
1011	1149	access_token=self.admin_user_tok,

1017	1155	self.assertEqual("bob", channel.json_body["displayname"])
1018	1156
1019	1157	# Get user
1020		request, channel = self.make_request(
1021		"GET", url, access_token=self.admin_user_tok,
1022		)
	1158	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1023	1159
1024	1160	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1025	1161	self.assertEqual("@bob:test", channel.json_body["name"])

1029	1165	# Change password (and use a str for deactivate instead of a bool)
1030	1166	body = json.dumps({"password": "abc123", "deactivated": "false"}) # oops!
1031	1167
1032		request, channel = self.make_request(
	1168	channel = self.make_request(
1033	1169	"PUT",
1034	1170	url,
1035	1171	access_token=self.admin_user_tok,

1039	1175	self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
1040	1176
1041	1177	# Check user is not deactivated
1042		request, channel = self.make_request(
1043		"GET", url, access_token=self.admin_user_tok,
1044		)
	1178	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1045	1179
1046	1180	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1047	1181	self.assertEqual("@bob:test", channel.json_body["name"])

1083	1217	"""
1084	1218	Try to list rooms of an user without authentication.
1085	1219	"""
1086		request, channel = self.make_request("GET", self.url, b"{}")
	1220	channel = self.make_request("GET", self.url, b"{}")
1087	1221
1088	1222	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1089	1223	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

1094	1228	"""
1095	1229	other_user_token = self.login("user", "pass")
1096	1230
1097		request, channel = self.make_request(
1098		"GET", self.url, access_token=other_user_token,
1099		)
	1231	channel = self.make_request("GET", self.url, access_token=other_user_token,)
1100	1232
1101	1233	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
1102	1234	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

1106	1238	Tests that a lookup for a user that does not exist returns a 404
1107	1239	"""
1108	1240	url = "/_synapse/admin/v1/users/@unknown_person:test/joined_rooms"
1109		request, channel = self.make_request(
1110		"GET", url, access_token=self.admin_user_tok,
1111		)
	1241	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1112	1242
1113	1243	self.assertEqual(404, channel.code, msg=channel.json_body)
1114	1244	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

1119	1249	"""
1120	1250	url = "/_synapse/admin/v1/users/@unknown_person:unknown_domain/joined_rooms"
1121	1251
1122		request, channel = self.make_request(
1123		"GET", url, access_token=self.admin_user_tok,
1124		)
	1252	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1125	1253
1126	1254	self.assertEqual(400, channel.code, msg=channel.json_body)
1127	1255	self.assertEqual("Can only lookup local users", channel.json_body["error"])

1132	1260	if user has no memberships
1133	1261	"""
1134	1262	# Get rooms
1135		request, channel = self.make_request(
1136		"GET", self.url, access_token=self.admin_user_tok,
1137		)
	1263	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1138	1264
1139	1265	self.assertEqual(200, channel.code, msg=channel.json_body)
1140	1266	self.assertEqual(0, channel.json_body["total"])

1151	1277	self.helper.create_room_as(self.other_user, tok=other_user_tok)
1152	1278
1153	1279	# Get rooms
1154		request, channel = self.make_request(
1155		"GET", self.url, access_token=self.admin_user_tok,
1156		)
	1280	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1157	1281
1158	1282	self.assertEqual(200, channel.code, msg=channel.json_body)
1159	1283	self.assertEqual(number_rooms, channel.json_body["total"])

1182	1306	"""
1183	1307	Try to list pushers of an user without authentication.
1184	1308	"""
1185		request, channel = self.make_request("GET", self.url, b"{}")
	1309	channel = self.make_request("GET", self.url, b"{}")
1186	1310
1187	1311	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1188	1312	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

1193	1317	"""
1194	1318	other_user_token = self.login("user", "pass")
1195	1319
1196		request, channel = self.make_request(
1197		"GET", self.url, access_token=other_user_token,
1198		)
	1320	channel = self.make_request("GET", self.url, access_token=other_user_token,)
1199	1321
1200	1322	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
1201	1323	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

1205	1327	Tests that a lookup for a user that does not exist returns a 404
1206	1328	"""
1207	1329	url = "/_synapse/admin/v1/users/@unknown_person:test/pushers"
1208		request, channel = self.make_request(
1209		"GET", url, access_token=self.admin_user_tok,
1210		)
	1330	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1211	1331
1212	1332	self.assertEqual(404, channel.code, msg=channel.json_body)
1213	1333	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

1218	1338	"""
1219	1339	url = "/_synapse/admin/v1/users/@unknown_person:unknown_domain/pushers"
1220	1340
1221		request, channel = self.make_request(
1222		"GET", url, access_token=self.admin_user_tok,
1223		)
	1341	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1224	1342
1225	1343	self.assertEqual(400, channel.code, msg=channel.json_body)
1226	1344	self.assertEqual("Can only lookup local users", channel.json_body["error"])

1231	1349	"""
1232	1350
1233	1351	# Get pushers
1234		request, channel = self.make_request(
1235		"GET", self.url, access_token=self.admin_user_tok,
1236		)
	1352	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1237	1353
1238	1354	self.assertEqual(200, channel.code, msg=channel.json_body)
1239	1355	self.assertEqual(0, channel.json_body["total"])

1255	1371	device_display_name="pushy push",
1256	1372	pushkey="a@example.com",
1257	1373	lang=None,
1258		data={"url": "example.com"},
	1374	data={"url": "https://example.com/_matrix/push/v1/notify"},
1259	1375	)
1260	1376	)
1261	1377
1262	1378	# Get pushers
1263		request, channel = self.make_request(
1264		"GET", self.url, access_token=self.admin_user_tok,
1265		)
	1379	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1266	1380
1267	1381	self.assertEqual(200, channel.code, msg=channel.json_body)
1268	1382	self.assertEqual(1, channel.json_body["total"])

1301	1415	"""
1302	1416	Try to list media of an user without authentication.
1303	1417	"""
1304		request, channel = self.make_request("GET", self.url, b"{}")
	1418	channel = self.make_request("GET", self.url, b"{}")
1305	1419
1306	1420	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1307	1421	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

1312	1426	"""
1313	1427	other_user_token = self.login("user", "pass")
1314	1428
1315		request, channel = self.make_request(
1316		"GET", self.url, access_token=other_user_token,
1317		)
	1429	channel = self.make_request("GET", self.url, access_token=other_user_token,)
1318	1430
1319	1431	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
1320	1432	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])

1324	1436	Tests that a lookup for a user that does not exist returns a 404
1325	1437	"""
1326	1438	url = "/_synapse/admin/v1/users/@unknown_person:test/media"
1327		request, channel = self.make_request(
1328		"GET", url, access_token=self.admin_user_tok,
1329		)
	1439	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1330	1440
1331	1441	self.assertEqual(404, channel.code, msg=channel.json_body)
1332	1442	self.assertEqual(Codes.NOT_FOUND, channel.json_body["errcode"])

1337	1447	"""
1338	1448	url = "/_synapse/admin/v1/users/@unknown_person:unknown_domain/media"
1339	1449
1340		request, channel = self.make_request(
1341		"GET", url, access_token=self.admin_user_tok,
1342		)
	1450	channel = self.make_request("GET", url, access_token=self.admin_user_tok,)
1343	1451
1344	1452	self.assertEqual(400, channel.code, msg=channel.json_body)
1345	1453	self.assertEqual("Can only lookup local users", channel.json_body["error"])

1353	1461	other_user_tok = self.login("user", "pass")
1354	1462	self._create_media(other_user_tok, number_media)
1355	1463
1356		request, channel = self.make_request(
	1464	channel = self.make_request(
1357	1465	"GET", self.url + "?limit=5", access_token=self.admin_user_tok,
1358	1466	)
1359	1467

1372	1480	other_user_tok = self.login("user", "pass")
1373	1481	self._create_media(other_user_tok, number_media)
1374	1482
1375		request, channel = self.make_request(
	1483	channel = self.make_request(
1376	1484	"GET", self.url + "?from=5", access_token=self.admin_user_tok,
1377	1485	)
1378	1486

1391	1499	other_user_tok = self.login("user", "pass")
1392	1500	self._create_media(other_user_tok, number_media)
1393	1501
1394		request, channel = self.make_request(
	1502	channel = self.make_request(
1395	1503	"GET", self.url + "?from=5&limit=10", access_token=self.admin_user_tok,
1396	1504	)
1397	1505

1406	1514	Testing that a negative limit parameter returns a 400
1407	1515	"""
1408	1516
1409		request, channel = self.make_request(
	1517	channel = self.make_request(
1410	1518	"GET", self.url + "?limit=-5", access_token=self.admin_user_tok,
1411	1519	)
1412	1520

1418	1526	Testing that a negative from parameter returns a 400
1419	1527	"""
1420	1528
1421		request, channel = self.make_request(
	1529	channel = self.make_request(
1422	1530	"GET", self.url + "?from=-5", access_token=self.admin_user_tok,
1423	1531	)
1424	1532

1436	1544
1437	1545	# `next_token` does not appear
1438	1546	# Number of results is the number of entries
1439		request, channel = self.make_request(
	1547	channel = self.make_request(
1440	1548	"GET", self.url + "?limit=20", access_token=self.admin_user_tok,
1441	1549	)
1442	1550

1447	1555
1448	1556	# `next_token` does not appear
1449	1557	# Number of max results is larger than the number of entries
1450		request, channel = self.make_request(
	1558	channel = self.make_request(
1451	1559	"GET", self.url + "?limit=21", access_token=self.admin_user_tok,
1452	1560	)
1453	1561

1458	1566
1459	1567	# `next_token` does appear
1460	1568	# Number of max results is smaller than the number of entries
1461		request, channel = self.make_request(
	1569	channel = self.make_request(
1462	1570	"GET", self.url + "?limit=19", access_token=self.admin_user_tok,
1463	1571	)
1464	1572

1470	1578	# Check
1471	1579	# Set `from` to value of `next_token` for request remaining entries
1472	1580	# `next_token` does not appear
1473		request, channel = self.make_request(
	1581	channel = self.make_request(
1474	1582	"GET", self.url + "?from=19", access_token=self.admin_user_tok,
1475	1583	)
1476	1584

1485	1593	if user has no media created
1486	1594	"""
1487	1595
1488		request, channel = self.make_request(
1489		"GET", self.url, access_token=self.admin_user_tok,
1490		)
	1596	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1491	1597
1492	1598	self.assertEqual(200, channel.code, msg=channel.json_body)
1493	1599	self.assertEqual(0, channel.json_body["total"])

1502	1608	other_user_tok = self.login("user", "pass")
1503	1609	self._create_media(other_user_tok, number_media)
1504	1610
1505		request, channel = self.make_request(
1506		"GET", self.url, access_token=self.admin_user_tok,
1507		)
	1611	channel = self.make_request("GET", self.url, access_token=self.admin_user_tok,)
1508	1612
1509	1613	self.assertEqual(200, channel.code, msg=channel.json_body)
1510	1614	self.assertEqual(number_media, channel.json_body["total"])

1570	1674	)
1571	1675
1572	1676	def _get_token(self) -> str:
1573		request, channel = self.make_request(
	1677	channel = self.make_request(
1574	1678	"POST", self.url, b"{}", access_token=self.admin_user_tok
1575	1679	)
1576	1680	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

1579	1683	def test_no_auth(self):
1580	1684	"""Try to login as a user without authentication.
1581	1685	"""
1582		request, channel = self.make_request("POST", self.url, b"{}")
	1686	channel = self.make_request("POST", self.url, b"{}")
1583	1687
1584	1688	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1585	1689	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])

1587	1691	def test_not_admin(self):
1588	1692	"""Try to login as a user as a non-admin user.
1589	1693	"""
1590		request, channel = self.make_request(
	1694	channel = self.make_request(
1591	1695	"POST", self.url, b"{}", access_token=self.other_user_tok
1592	1696	)
1593	1697

1615	1719	self._get_token()
1616	1720
1617	1721	# Check that we don't see a new device in our devices list
1618		request, channel = self.make_request(
	1722	channel = self.make_request(
1619	1723	"GET", "devices", b"{}", access_token=self.other_user_tok
1620	1724	)
1621	1725	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

1630	1734	puppet_token = self._get_token()
1631	1735
1632	1736	# Test that we can successfully make a request
1633		request, channel = self.make_request(
1634		"GET", "devices", b"{}", access_token=puppet_token
1635		)
	1737	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1636	1738	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1637	1739
1638	1740	# Logout with the puppet token
1639		request, channel = self.make_request(
1640		"POST", "logout", b"{}", access_token=puppet_token
1641		)
	1741	channel = self.make_request("POST", "logout", b"{}", access_token=puppet_token)
1642	1742	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1643	1743
1644	1744	# The puppet token should no longer work
1645		request, channel = self.make_request(
1646		"GET", "devices", b"{}", access_token=puppet_token
1647		)
	1745	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1648	1746	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1649	1747
1650	1748	# .. but the real user's tokens should still work
1651		request, channel = self.make_request(
	1749	channel = self.make_request(
1652	1750	"GET", "devices", b"{}", access_token=self.other_user_tok
1653	1751	)
1654	1752	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

1661	1759	puppet_token = self._get_token()
1662	1760
1663	1761	# Test that we can successfully make a request
1664		request, channel = self.make_request(
1665		"GET", "devices", b"{}", access_token=puppet_token
1666		)
	1762	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1667	1763	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1668	1764
1669	1765	# Logout all with the real user token
1670		request, channel = self.make_request(
	1766	channel = self.make_request(
1671	1767	"POST", "logout/all", b"{}", access_token=self.other_user_tok
1672	1768	)
1673	1769	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1674	1770
1675	1771	# The puppet token should still work
1676		request, channel = self.make_request(
1677		"GET", "devices", b"{}", access_token=puppet_token
1678		)
	1772	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1679	1773	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1680	1774
1681	1775	# .. but the real user's tokens shouldn't
1682		request, channel = self.make_request(
	1776	channel = self.make_request(
1683	1777	"GET", "devices", b"{}", access_token=self.other_user_tok
1684	1778	)
1685	1779	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])

1692	1786	puppet_token = self._get_token()
1693	1787
1694	1788	# Test that we can successfully make a request
1695		request, channel = self.make_request(
1696		"GET", "devices", b"{}", access_token=puppet_token
1697		)
	1789	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1698	1790	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1699	1791
1700	1792	# Logout all with the admin user token
1701		request, channel = self.make_request(
	1793	channel = self.make_request(
1702	1794	"POST", "logout/all", b"{}", access_token=self.admin_user_tok
1703	1795	)
1704	1796	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
1705	1797
1706	1798	# The puppet token should no longer work
1707		request, channel = self.make_request(
1708		"GET", "devices", b"{}", access_token=puppet_token
1709		)
	1799	channel = self.make_request("GET", "devices", b"{}", access_token=puppet_token)
1710	1800	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1711	1801
1712	1802	# .. but the real user's tokens should still work
1713		request, channel = self.make_request(
	1803	channel = self.make_request(
1714	1804	"GET", "devices", b"{}", access_token=self.other_user_tok
1715	1805	)
1716	1806	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])

1792	1882	"""
1793	1883	Try to get information of an user without authentication.
1794	1884	"""
1795		request, channel = self.make_request("GET", self.url1, b"{}")
	1885	channel = self.make_request("GET", self.url1, b"{}")
1796	1886	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1797	1887	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
1798	1888
1799		request, channel = self.make_request("GET", self.url2, b"{}")
	1889	channel = self.make_request("GET", self.url2, b"{}")
1800	1890	self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
1801	1891	self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
1802	1892

1807	1897	self.register_user("user2", "pass")
1808	1898	other_user2_token = self.login("user2", "pass")
1809	1899
1810		request, channel = self.make_request(
1811		"GET", self.url1, access_token=other_user2_token,
1812		)
	1900	channel = self.make_request("GET", self.url1, access_token=other_user2_token,)
1813	1901	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
1814	1902	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
1815	1903
1816		request, channel = self.make_request(
1817		"GET", self.url2, access_token=other_user2_token,
1818		)
	1904	channel = self.make_request("GET", self.url2, access_token=other_user2_token,)
1819	1905	self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
1820	1906	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
1821	1907

1826	1912	url1 = "/_synapse/admin/v1/whois/@unknown_person:unknown_domain"
1827	1913	url2 = "/_matrix/client/r0/admin/whois/@unknown_person:unknown_domain"
1828	1914
1829		request, channel = self.make_request(
1830		"GET", url1, access_token=self.admin_user_tok,
1831		)
	1915	channel = self.make_request("GET", url1, access_token=self.admin_user_tok,)
1832	1916	self.assertEqual(400, channel.code, msg=channel.json_body)
1833	1917	self.assertEqual("Can only whois a local user", channel.json_body["error"])
1834	1918
1835		request, channel = self.make_request(
1836		"GET", url2, access_token=self.admin_user_tok,
1837		)
	1919	channel = self.make_request("GET", url2, access_token=self.admin_user_tok,)
1838	1920	self.assertEqual(400, channel.code, msg=channel.json_body)
1839	1921	self.assertEqual("Can only whois a local user", channel.json_body["error"])
1840	1922

1842	1924	"""
1843	1925	The lookup should succeed for an admin.
1844	1926	"""
1845		request, channel = self.make_request(
1846		"GET", self.url1, access_token=self.admin_user_tok,
1847		)
	1927	channel = self.make_request("GET", self.url1, access_token=self.admin_user_tok,)
1848	1928	self.assertEqual(200, channel.code, msg=channel.json_body)
1849	1929	self.assertEqual(self.other_user, channel.json_body["user_id"])
1850	1930	self.assertIn("devices", channel.json_body)
1851	1931
1852		request, channel = self.make_request(
1853		"GET", self.url2, access_token=self.admin_user_tok,
1854		)
	1932	channel = self.make_request("GET", self.url2, access_token=self.admin_user_tok,)
1855	1933	self.assertEqual(200, channel.code, msg=channel.json_body)
1856	1934	self.assertEqual(self.other_user, channel.json_body["user_id"])
1857	1935	self.assertIn("devices", channel.json_body)

1862	1940	"""
1863	1941	other_user_token = self.login("user", "pass")
1864	1942
1865		request, channel = self.make_request(
1866		"GET", self.url1, access_token=other_user_token,
1867		)
	1943	channel = self.make_request("GET", self.url1, access_token=other_user_token,)
1868	1944	self.assertEqual(200, channel.code, msg=channel.json_body)
1869	1945	self.assertEqual(self.other_user, channel.json_body["user_id"])
1870	1946	self.assertIn("devices", channel.json_body)
1871	1947
1872		request, channel = self.make_request(
1873		"GET", self.url2, access_token=other_user_token,
1874		)
	1948	channel = self.make_request("GET", self.url2, access_token=other_user_token,)
1875	1949	self.assertEqual(200, channel.code, msg=channel.json_body)
1876	1950	self.assertEqual(self.other_user, channel.json_body["user_id"])
1877	1951	self.assertIn("devices", channel.json_body)

+4

-4

tests/rest/client/test_consent.py less more

60	60	def test_render_public_consent(self):
61	61	"""You can observe the terms form without specifying a user"""
62	62	resource = consent_resource.ConsentResource(self.hs)
63		request, channel = make_request(
	63	channel = make_request(
64	64	self.reactor, FakeSite(resource), "GET", "/consent?v=1", shorthand=False
65	65	)
66	66	self.assertEqual(channel.code, 200)

81	81	uri_builder.build_user_consent_uri(user_id).replace("_matrix/", "")
82	82	+ "&u=user"
83	83	)
84		request, channel = make_request(
	84	channel = make_request(
85	85	self.reactor,
86	86	FakeSite(resource),
87	87	"GET",

96	96	self.assertEqual(consented, "False")
97	97
98	98	# POST to the consent page, saying we've agreed
99		request, channel = make_request(
	99	channel = make_request(
100	100	self.reactor,
101	101	FakeSite(resource),
102	102	"POST",

108	108
109	109	# Fetch the consent page, to get the consent version -- it should have
110	110	# changed
111		request, channel = make_request(
	111	channel = make_request(
112	112	self.reactor,
113	113	FakeSite(resource),
114	114	"GET",

+1

-1

tests/rest/client/test_ephemeral_message.py less more

92	92	def get_event(self, room_id, event_id, expected_code=200):
93	93	url = "/_matrix/client/r0/rooms/%s/event/%s" % (room_id, event_id)
94	94
95		request, channel = self.make_request("GET", url)
	95	channel = self.make_request("GET", url)
96	96
97	97	self.assertEqual(channel.code, expected_code, channel.result)
98	98

+2

-4

tests/rest/client/test_identity.py less more

42	42	self.register_user("kermit", "monkey")
43	43	tok = self.login("kermit", "monkey")
44	44
45		request, channel = self.make_request(
46		b"POST", "/createRoom", b"{}", access_token=tok
47		)
	45	channel = self.make_request(b"POST", "/createRoom", b"{}", access_token=tok)
48	46	self.assertEquals(channel.result["code"], b"200", channel.result)
49	47	room_id = channel.json_body["room_id"]
50	48

55	53	}
56	54	request_data = json.dumps(params)
57	55	request_url = ("/rooms/%s/invite" % (room_id)).encode("ascii")
58		request, channel = self.make_request(
	56	channel = self.make_request(
59	57	b"POST", request_url, request_data, access_token=tok
60	58	)
61	59	self.assertEquals(channel.result["code"], b"403", channel.result)

+2

-6

tests/rest/client/test_redactions.py less more

68	68	"""
69	69	path = "/_matrix/client/r0/rooms/%s/redact/%s" % (room_id, event_id)
70	70
71		request, channel = self.make_request(
72		"POST", path, content={}, access_token=access_token
73		)
	71	channel = self.make_request("POST", path, content={}, access_token=access_token)
74	72	self.assertEqual(int(channel.result["code"]), expect_code)
75	73	return channel.json_body
76	74
77	75	def _sync_room_timeline(self, access_token, room_id):
78		request, channel = self.make_request(
79		"GET", "sync", access_token=self.mod_access_token
80		)
	76	channel = self.make_request("GET", "sync", access_token=self.mod_access_token)
81	77	self.assertEqual(channel.result["code"], b"200")
82	78	room_sync = channel.json_body["rooms"]["join"][room_id]
83	79	return room_sync["timeline"]["events"]

+1

-1

tests/rest/client/test_retention.py less more

324	324	def get_event(self, room_id, event_id, expected_code=200):
325	325	url = "/_matrix/client/r0/rooms/%s/event/%s" % (room_id, event_id)
326	326
327		request, channel = self.make_request("GET", url, access_token=self.token)
	327	channel = self.make_request("GET", url, access_token=self.token)
328	328
329	329	self.assertEqual(channel.code, expected_code, channel.result)
330	330

+8

-8

tests/rest/client/test_shadow_banned.py less more

88	88	)
89	89
90	90	# Inviting the user completes successfully.
91		request, channel = self.make_request(
	91	channel = self.make_request(
92	92	"POST",
93	93	"/rooms/%s/invite" % (room_id,),
94	94	{"id_server": "test", "medium": "email", "address": "test@test.test"},

102	102	def test_create_room(self):
103	103	"""Invitations during a room creation should be discarded, but the room still gets created."""
104	104	# The room creation is successful.
105		request, channel = self.make_request(
	105	channel = self.make_request(
106	106	"POST",
107	107	"/_matrix/client/r0/createRoom",
108	108	{"visibility": "public", "invite": [self.other_user_id]},

157	157	self.banned_user_id, tok=self.banned_access_token
158	158	)
159	159
160		request, channel = self.make_request(
	160	channel = self.make_request(
161	161	"POST",
162	162	"/_matrix/client/r0/rooms/%s/upgrade" % (room_id,),
163	163	{"new_version": "6"},

182	182	self.banned_user_id, tok=self.banned_access_token
183	183	)
184	184
185		request, channel = self.make_request(
	185	channel = self.make_request(
186	186	"PUT",
187	187	"/rooms/%s/typing/%s" % (room_id, self.banned_user_id),
188	188	{"typing": True, "timeout": 30000},

197	197	# The other user can join and send typing events.
198	198	self.helper.join(room_id, self.other_user_id, tok=self.other_access_token)
199	199
200		request, channel = self.make_request(
	200	channel = self.make_request(
201	201	"PUT",
202	202	"/rooms/%s/typing/%s" % (room_id, self.other_user_id),
203	203	{"typing": True, "timeout": 30000},

243	243	)
244	244
245	245	# The update should succeed.
246		request, channel = self.make_request(
	246	channel = self.make_request(
247	247	"PUT",
248	248	"/_matrix/client/r0/profile/%s/displayname" % (self.banned_user_id,),
249	249	{"displayname": new_display_name},

253	253	self.assertEqual(channel.json_body, {})
254	254
255	255	# The user's display name should be updated.
256		request, channel = self.make_request(
	256	channel = self.make_request(
257	257	"GET", "/profile/%s/displayname" % (self.banned_user_id,)
258	258	)
259	259	self.assertEqual(channel.code, 200, channel.result)

281	281	)
282	282
283	283	# The update should succeed.
284		request, channel = self.make_request(
	284	channel = self.make_request(
285	285	"PUT",
286	286	"/_matrix/client/r0/rooms/%s/state/m.room.member/%s"
287	287	% (room_id, self.banned_user_id),

+5

-5

tests/rest/client/test_third_party_rules.py less more

85	85	callback = Mock(spec=[], side_effect=check)
86	86	current_rules_module().check_event_allowed = callback
87	87
88		request, channel = self.make_request(
	88	channel = self.make_request(
89	89	"PUT",
90	90	"/_matrix/client/r0/rooms/%s/send/foo.bar.allowed/1" % self.room_id,
91	91	{},

103	103	self.assertEqual(ev.type, k[0])
104	104	self.assertEqual(ev.state_key, k[1])
105	105
106		request, channel = self.make_request(
	106	channel = self.make_request(
107	107	"PUT",
108	108	"/_matrix/client/r0/rooms/%s/send/foo.bar.forbidden/2" % self.room_id,
109	109	{},

122	122	current_rules_module().check_event_allowed = check
123	123
124	124	# now send the event
125		request, channel = self.make_request(
	125	channel = self.make_request(
126	126	"PUT",
127	127	"/_matrix/client/r0/rooms/%s/send/modifyme/1" % self.room_id,
128	128	{"x": "x"},

141	141	current_rules_module().check_event_allowed = check
142	142
143	143	# now send the event
144		request, channel = self.make_request(
	144	channel = self.make_request(
145	145	"PUT",
146	146	"/_matrix/client/r0/rooms/%s/send/modifyme/1" % self.room_id,
147	147	{"x": "x"},

151	151	event_id = channel.json_body["event_id"]
152	152
153	153	# ... and check that it got modified
154		request, channel = self.make_request(
	154	channel = self.make_request(
155	155	"GET",
156	156	"/_matrix/client/r0/rooms/%s/event/%s" % (self.room_id, event_id),
157	157	access_token=self.tok,

+4

-4

tests/rest/client/v1/test_directory.py less more

90	90	# that we can make sure that the check is done on the whole alias.
91	91	data = {"room_alias_name": random_string(256 - len(self.hs.hostname))}
92	92	request_data = json.dumps(data)
93		request, channel = self.make_request(
	93	channel = self.make_request(
94	94	"POST", url, request_data, access_token=self.user_tok
95	95	)
96	96	self.assertEqual(channel.code, 400, channel.result)

103	103	# as cautious as possible here.
104	104	data = {"room_alias_name": random_string(5)}
105	105	request_data = json.dumps(data)
106		request, channel = self.make_request(
	106	channel = self.make_request(
107	107	"POST", url, request_data, access_token=self.user_tok
108	108	)
109	109	self.assertEqual(channel.code, 200, channel.result)

117	117	data = {"aliases": [self.random_alias(alias_length)]}
118	118	request_data = json.dumps(data)
119	119
120		request, channel = self.make_request(
	120	channel = self.make_request(
121	121	"PUT", url, request_data, access_token=self.user_tok
122	122	)
123	123	self.assertEqual(channel.code, expected_code, channel.result)

127	127	data = {"room_id": self.room_id}
128	128	request_data = json.dumps(data)
129	129
130		request, channel = self.make_request(
	130	channel = self.make_request(
131	131	"PUT", url, request_data, access_token=self.user_tok
132	132	)
133	133	self.assertEqual(channel.code, expected_code, channel.result)

+4

-4

tests/rest/client/v1/test_events.py less more

62	62	# implementation is now part of the r0 implementation, the newer
63	63	# behaviour is used instead to be consistent with the r0 spec.
64	64	# see issue #2602
65		request, channel = self.make_request(
	65	channel = self.make_request(
66	66	"GET", "/events?access_token=%s" % ("invalid" + self.token,)
67	67	)
68	68	self.assertEquals(channel.code, 401, msg=channel.result)
69	69
70	70	# valid token, expect content
71		request, channel = self.make_request(
	71	channel = self.make_request(
72	72	"GET", "/events?access_token=%s&timeout=0" % (self.token,)
73	73	)
74	74	self.assertEquals(channel.code, 200, msg=channel.result)

86	86	)
87	87
88	88	# valid token, expect content
89		request, channel = self.make_request(
	89	channel = self.make_request(
90	90	"GET", "/events?access_token=%s&timeout=0" % (self.token,)
91	91	)
92	92	self.assertEquals(channel.code, 200, msg=channel.result)

148	148	resp = self.helper.send(self.room_id, tok=self.token)
149	149	event_id = resp["event_id"]
150	150
151		request, channel = self.make_request(
	151	channel = self.make_request(
152	152	"GET", "/events/" + event_id, access_token=self.token,
153	153	)
154	154	self.assertEquals(channel.code, 200, msg=channel.result)

+44

-56

tests/rest/client/v1/test_login.py less more

62	62	"identifier": {"type": "m.id.user", "user": "kermit" + str(i)},
63	63	"password": "monkey",
64	64	}
65		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	65	channel = self.make_request(b"POST", LOGIN_URL, params)
66	66
67	67	if i == 5:
68	68	self.assertEquals(channel.result["code"], b"429", channel.result)

81	81	"identifier": {"type": "m.id.user", "user": "kermit" + str(i)},
82	82	"password": "monkey",
83	83	}
84		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	84	channel = self.make_request(b"POST", LOGIN_URL, params)
85	85
86	86	self.assertEquals(channel.result["code"], b"200", channel.result)
87	87

107	107	"identifier": {"type": "m.id.user", "user": "kermit"},
108	108	"password": "monkey",
109	109	}
110		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	110	channel = self.make_request(b"POST", LOGIN_URL, params)
111	111
112	112	if i == 5:
113	113	self.assertEquals(channel.result["code"], b"429", channel.result)

126	126	"identifier": {"type": "m.id.user", "user": "kermit"},
127	127	"password": "monkey",
128	128	}
129		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	129	channel = self.make_request(b"POST", LOGIN_URL, params)
130	130
131	131	self.assertEquals(channel.result["code"], b"200", channel.result)
132	132

152	152	"identifier": {"type": "m.id.user", "user": "kermit"},
153	153	"password": "notamonkey",
154	154	}
155		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	155	channel = self.make_request(b"POST", LOGIN_URL, params)
156	156
157	157	if i == 5:
158	158	self.assertEquals(channel.result["code"], b"429", channel.result)

171	171	"identifier": {"type": "m.id.user", "user": "kermit"},
172	172	"password": "notamonkey",
173	173	}
174		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	174	channel = self.make_request(b"POST", LOGIN_URL, params)
175	175
176	176	self.assertEquals(channel.result["code"], b"403", channel.result)
177	177

180	180	self.register_user("kermit", "monkey")
181	181
182	182	# we shouldn't be able to make requests without an access token
183		request, channel = self.make_request(b"GET", TEST_URL)
	183	channel = self.make_request(b"GET", TEST_URL)
184	184	self.assertEquals(channel.result["code"], b"401", channel.result)
185	185	self.assertEquals(channel.json_body["errcode"], "M_MISSING_TOKEN")
186	186

190	190	"identifier": {"type": "m.id.user", "user": "kermit"},
191	191	"password": "monkey",
192	192	}
193		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	193	channel = self.make_request(b"POST", LOGIN_URL, params)
194	194
195	195	self.assertEquals(channel.code, 200, channel.result)
196	196	access_token = channel.json_body["access_token"]
197	197	device_id = channel.json_body["device_id"]
198	198
199	199	# we should now be able to make requests with the access token
200		request, channel = self.make_request(
201		b"GET", TEST_URL, access_token=access_token
202		)
	200	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
203	201	self.assertEquals(channel.code, 200, channel.result)
204	202
205	203	# time passes
206	204	self.reactor.advance(24 * 3600)
207	205
208	206	# ... and we should be soft-logouted
209		request, channel = self.make_request(
210		b"GET", TEST_URL, access_token=access_token
211		)
	207	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
212	208	self.assertEquals(channel.code, 401, channel.result)
213	209	self.assertEquals(channel.json_body["errcode"], "M_UNKNOWN_TOKEN")
214	210	self.assertEquals(channel.json_body["soft_logout"], True)

222	218
223	219	# more requests with the expired token should still return a soft-logout
224	220	self.reactor.advance(3600)
225		request, channel = self.make_request(
226		b"GET", TEST_URL, access_token=access_token
227		)
	221	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
228	222	self.assertEquals(channel.code, 401, channel.result)
229	223	self.assertEquals(channel.json_body["errcode"], "M_UNKNOWN_TOKEN")
230	224	self.assertEquals(channel.json_body["soft_logout"], True)

232	226	# ... but if we delete that device, it will be a proper logout
233	227	self._delete_device(access_token_2, "kermit", "monkey", device_id)
234	228
235		request, channel = self.make_request(
236		b"GET", TEST_URL, access_token=access_token
237		)
	229	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
238	230	self.assertEquals(channel.code, 401, channel.result)
239	231	self.assertEquals(channel.json_body["errcode"], "M_UNKNOWN_TOKEN")
240	232	self.assertEquals(channel.json_body["soft_logout"], False)
241	233
242	234	def _delete_device(self, access_token, user_id, password, device_id):
243	235	"""Perform the UI-Auth to delete a device"""
244		request, channel = self.make_request(
	236	channel = self.make_request(
245	237	b"DELETE", "devices/" + device_id, access_token=access_token
246	238	)
247	239	self.assertEquals(channel.code, 401, channel.result)

261	253	"session": channel.json_body["session"],
262	254	}
263	255
264		request, channel = self.make_request(
	256	channel = self.make_request(
265	257	b"DELETE",
266	258	"devices/" + device_id,
267	259	access_token=access_token,

277	269	access_token = self.login("kermit", "monkey")
278	270
279	271	# we should now be able to make requests with the access token
280		request, channel = self.make_request(
281		b"GET", TEST_URL, access_token=access_token
282		)
	272	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
283	273	self.assertEquals(channel.code, 200, channel.result)
284	274
285	275	# time passes
286	276	self.reactor.advance(24 * 3600)
287	277
288	278	# ... and we should be soft-logouted
289		request, channel = self.make_request(
290		b"GET", TEST_URL, access_token=access_token
291		)
	279	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
292	280	self.assertEquals(channel.code, 401, channel.result)
293	281	self.assertEquals(channel.json_body["errcode"], "M_UNKNOWN_TOKEN")
294	282	self.assertEquals(channel.json_body["soft_logout"], True)
295	283
296	284	# Now try to hard logout this session
297		request, channel = self.make_request(
298		b"POST", "/logout", access_token=access_token
299		)
	285	channel = self.make_request(b"POST", "/logout", access_token=access_token)
300	286	self.assertEquals(channel.result["code"], b"200", channel.result)
301	287
302	288	@override_config({"session_lifetime": "24h"})

307	293	access_token = self.login("kermit", "monkey")
308	294
309	295	# we should now be able to make requests with the access token
310		request, channel = self.make_request(
311		b"GET", TEST_URL, access_token=access_token
312		)
	296	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
313	297	self.assertEquals(channel.code, 200, channel.result)
314	298
315	299	# time passes
316	300	self.reactor.advance(24 * 3600)
317	301
318	302	# ... and we should be soft-logouted
319		request, channel = self.make_request(
320		b"GET", TEST_URL, access_token=access_token
321		)
	303	channel = self.make_request(b"GET", TEST_URL, access_token=access_token)
322	304	self.assertEquals(channel.code, 401, channel.result)
323	305	self.assertEquals(channel.json_body["errcode"], "M_UNKNOWN_TOKEN")
324	306	self.assertEquals(channel.json_body["soft_logout"], True)
325	307
326	308	# Now try to hard log out all of the user's sessions
327		request, channel = self.make_request(
328		b"POST", "/logout/all", access_token=access_token
329		)
	309	channel = self.make_request(b"POST", "/logout/all", access_token=access_token)
330	310	self.assertEquals(channel.result["code"], b"200", channel.result)
331	311
332	312

401	381	cas_ticket_url = urllib.parse.urlunparse(url_parts)
402	382
403	383	# Get Synapse to call the fake CAS and serve the template.
404		request, channel = self.make_request("GET", cas_ticket_url)
	384	channel = self.make_request("GET", cas_ticket_url)
405	385
406	386	# Test that the response is HTML.
407	387	self.assertEqual(channel.code, 200)

445	425	)
446	426
447	427	# Get Synapse to call the fake CAS and serve the template.
448		request, channel = self.make_request("GET", cas_ticket_url)
	428	channel = self.make_request("GET", cas_ticket_url)
449	429
450	430	self.assertEqual(channel.code, 302)
451	431	location_headers = channel.headers.getRawHeaders("Location")

471	451	)
472	452
473	453	# Get Synapse to call the fake CAS and serve the template.
474		request, channel = self.make_request("GET", cas_ticket_url)
	454	channel = self.make_request("GET", cas_ticket_url)
475	455
476	456	# Because the user is deactivated they are served an error template.
477	457	self.assertEqual(channel.code, 403)

494	474	self.hs.config.jwt_algorithm = self.jwt_algorithm
495	475	return self.hs
496	476
497		def jwt_encode(self, token, secret=jwt_secret):
498		return jwt.encode(token, secret, self.jwt_algorithm).decode("ascii")
	477	def jwt_encode(self, token: str, secret: str = jwt_secret) -> str:
	478	# PyJWT 2.0.0 changed the return type of jwt.encode from bytes to str.
	479	result = jwt.encode(token, secret, self.jwt_algorithm)
	480	if isinstance(result, bytes):
	481	return result.decode("ascii")
	482	return result
499	483
500	484	def jwt_login(self, *args):
501	485	params = json.dumps(
502	486	{"type": "org.matrix.login.jwt", "token": self.jwt_encode(*args)}
503	487	)
504		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	488	channel = self.make_request(b"POST", LOGIN_URL, params)
505	489	return channel
506	490
507	491	def test_login_jwt_valid_registered(self):

633	617
634	618	def test_login_no_token(self):
635	619	params = json.dumps({"type": "org.matrix.login.jwt"})
636		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	620	channel = self.make_request(b"POST", LOGIN_URL, params)
637	621	self.assertEqual(channel.result["code"], b"403", channel.result)
638	622	self.assertEqual(channel.json_body["errcode"], "M_FORBIDDEN")
639	623	self.assertEqual(channel.json_body["error"], "Token field for JWT is missing")

699	683	self.hs.config.jwt_algorithm = "RS256"
700	684	return self.hs
701	685
702		def jwt_encode(self, token, secret=jwt_privatekey):
703		return jwt.encode(token, secret, "RS256").decode("ascii")
	686	def jwt_encode(self, token: str, secret: str = jwt_privatekey) -> str:
	687	# PyJWT 2.0.0 changed the return type of jwt.encode from bytes to str.
	688	result = jwt.encode(token, secret, "RS256")
	689	if isinstance(result, bytes):
	690	return result.decode("ascii")
	691	return result
704	692
705	693	def jwt_login(self, *args):
706	694	params = json.dumps(
707	695	{"type": "org.matrix.login.jwt", "token": self.jwt_encode(*args)}
708	696	)
709		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	697	channel = self.make_request(b"POST", LOGIN_URL, params)
710	698	return channel
711	699
712	700	def test_login_jwt_valid(self):

734	722	]
735	723
736	724	def register_as_user(self, username):
737		request, channel = self.make_request(
	725	self.make_request(
738	726	b"POST",
739	727	"/_matrix/client/r0/register?access_token=%s" % (self.service.token,),
740	728	{"username": username},

783	771	"type": login.LoginRestServlet.APPSERVICE_TYPE,
784	772	"identifier": {"type": "m.id.user", "user": AS_USER},
785	773	}
786		request, channel = self.make_request(
	774	channel = self.make_request(
787	775	b"POST", LOGIN_URL, params, access_token=self.service.token
788	776	)
789	777

798	786	"type": login.LoginRestServlet.APPSERVICE_TYPE,
799	787	"identifier": {"type": "m.id.user", "user": self.service.sender},
800	788	}
801		request, channel = self.make_request(
	789	channel = self.make_request(
802	790	b"POST", LOGIN_URL, params, access_token=self.service.token
803	791	)
804	792

813	801	"type": login.LoginRestServlet.APPSERVICE_TYPE,
814	802	"identifier": {"type": "m.id.user", "user": "fibble_wibble"},
815	803	}
816		request, channel = self.make_request(
	804	channel = self.make_request(
817	805	b"POST", LOGIN_URL, params, access_token=self.service.token
818	806	)
819	807

828	816	"type": login.LoginRestServlet.APPSERVICE_TYPE,
829	817	"identifier": {"type": "m.id.user", "user": AS_USER},
830	818	}
831		request, channel = self.make_request(
	819	channel = self.make_request(
832	820	b"POST", LOGIN_URL, params, access_token=self.another_service.token
833	821	)
834	822

844	832	"type": login.LoginRestServlet.APPSERVICE_TYPE,
845	833	"identifier": {"type": "m.id.user", "user": AS_USER},
846	834	}
847		request, channel = self.make_request(b"POST", LOGIN_URL, params)
	835	channel = self.make_request(b"POST", LOGIN_URL, params)
848	836
849	837	self.assertEquals(channel.result["code"], b"401", channel.result)

+3

-3

tests/rest/client/v1/test_presence.py less more

37	37
38	38	hs = self.setup_test_homeserver(
39	39	"red",
40		http_client=None,
	40	federation_http_client=None,
41	41	federation_client=Mock(),
42	42	presence_handler=presence_handler,
43	43	)

52	52	self.hs.config.use_presence = True
53	53
54	54	body = {"presence": "here", "status_msg": "beep boop"}
55		request, channel = self.make_request(
	55	channel = self.make_request(
56	56	"PUT", "/presence/%s/status" % (self.user_id,), body
57	57	)
58	58

67	67	self.hs.config.use_presence = False
68	68
69	69	body = {"presence": "here", "status_msg": "beep boop"}
70		request, channel = self.make_request(
	70	channel = self.make_request(
71	71	"PUT", "/presence/%s/status" % (self.user_id,), body
72	72	)
73	73

+8

-10

tests/rest/client/v1/test_profile.py less more

62	62	hs = yield setup_test_homeserver(
63	63	self.addCleanup,
64	64	"test",
65		http_client=None,
	65	federation_http_client=None,
66	66	resource_for_client=self.mock_resource,
67	67	federation=Mock(),
68	68	federation_client=Mock(),

188	188	self.owner_tok = self.login("owner", "pass")
189	189
190	190	def test_set_displayname(self):
191		request, channel = self.make_request(
	191	channel = self.make_request(
192	192	"PUT",
193	193	"/profile/%s/displayname" % (self.owner,),
194	194	content=json.dumps({"displayname": "test"}),

201	201
202	202	def test_set_displayname_too_long(self):
203	203	"""Attempts to set a stupid displayname should get a 400"""
204		request, channel = self.make_request(
	204	channel = self.make_request(
205	205	"PUT",
206	206	"/profile/%s/displayname" % (self.owner,),
207	207	content=json.dumps({"displayname": "test" * 100}),

213	213	self.assertEqual(res, "owner")
214	214
215	215	def get_displayname(self):
216		request, channel = self.make_request(
217		"GET", "/profile/%s/displayname" % (self.owner,)
218		)
	216	channel = self.make_request("GET", "/profile/%s/displayname" % (self.owner,))
219	217	self.assertEqual(channel.code, 200, channel.result)
220	218	return channel.json_body["displayname"]
221	219

277	275	)
278	276
279	277	def request_profile(self, expected_code, url_suffix="", access_token=None):
280		request, channel = self.make_request(
	278	channel = self.make_request(
281	279	"GET", self.profile_url + url_suffix, access_token=access_token
282	280	)
283	281	self.assertEqual(channel.code, expected_code, channel.result)

319	317	"""Tests that a user can lookup their own profile without having to be in a room
320	318	if 'require_auth_for_profile_requests' is set to true in the server's config.
321	319	"""
322		request, channel = self.make_request(
	320	channel = self.make_request(
323	321	"GET", "/profile/" + self.requester, access_token=self.requester_tok
324	322	)
325	323	self.assertEqual(channel.code, 200, channel.result)
326	324
327		request, channel = self.make_request(
	325	channel = self.make_request(
328	326	"GET",
329	327	"/profile/" + self.requester + "/displayname",
330	328	access_token=self.requester_tok,
331	329	)
332	330	self.assertEqual(channel.code, 200, channel.result)
333	331
334		request, channel = self.make_request(
	332	channel = self.make_request(
335	333	"GET",
336	334	"/profile/" + self.requester + "/avatar_url",
337	335	access_token=self.requester_tok,

+45

-45

tests/rest/client/v1/test_push_rule_attrs.py less more

44	44	}
45	45
46	46	# PUT a new rule
47		request, channel = self.make_request(
	47	channel = self.make_request(
48	48	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
49	49	)
50	50	self.assertEqual(channel.code, 200)
51	51
52	52	# GET enabled for that new rule
53		request, channel = self.make_request(
	53	channel = self.make_request(
54	54	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
55	55	)
56	56	self.assertEqual(channel.code, 200)

73	73	}
74	74
75	75	# PUT a new rule
76		request, channel = self.make_request(
	76	channel = self.make_request(
77	77	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
78	78	)
79	79	self.assertEqual(channel.code, 200)
80	80
81	81	# disable the rule
82		request, channel = self.make_request(
	82	channel = self.make_request(
83	83	"PUT",
84	84	"/pushrules/global/override/best.friend/enabled",
85	85	{"enabled": False},

88	88	self.assertEqual(channel.code, 200)
89	89
90	90	# check rule disabled
91		request, channel = self.make_request(
	91	channel = self.make_request(
92	92	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
93	93	)
94	94	self.assertEqual(channel.code, 200)
95	95	self.assertEqual(channel.json_body["enabled"], False)
96	96
97	97	# DELETE the rule
98		request, channel = self.make_request(
	98	channel = self.make_request(
99	99	"DELETE", "/pushrules/global/override/best.friend", access_token=token
100	100	)
101	101	self.assertEqual(channel.code, 200)
102	102
103	103	# PUT a new rule
104		request, channel = self.make_request(
	104	channel = self.make_request(
105	105	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
106	106	)
107	107	self.assertEqual(channel.code, 200)
108	108
109	109	# GET enabled for that new rule
110		request, channel = self.make_request(
	110	channel = self.make_request(
111	111	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
112	112	)
113	113	self.assertEqual(channel.code, 200)

129	129	}
130	130
131	131	# PUT a new rule
132		request, channel = self.make_request(
	132	channel = self.make_request(
133	133	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
134	134	)
135	135	self.assertEqual(channel.code, 200)
136	136
137	137	# disable the rule
138		request, channel = self.make_request(
	138	channel = self.make_request(
139	139	"PUT",
140	140	"/pushrules/global/override/best.friend/enabled",
141	141	{"enabled": False},

144	144	self.assertEqual(channel.code, 200)
145	145
146	146	# check rule disabled
147		request, channel = self.make_request(
	147	channel = self.make_request(
148	148	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
149	149	)
150	150	self.assertEqual(channel.code, 200)
151	151	self.assertEqual(channel.json_body["enabled"], False)
152	152
153	153	# re-enable the rule
154		request, channel = self.make_request(
	154	channel = self.make_request(
155	155	"PUT",
156	156	"/pushrules/global/override/best.friend/enabled",
157	157	{"enabled": True},

160	160	self.assertEqual(channel.code, 200)
161	161
162	162	# check rule enabled
163		request, channel = self.make_request(
	163	channel = self.make_request(
164	164	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
165	165	)
166	166	self.assertEqual(channel.code, 200)

181	181	}
182	182
183	183	# check 404 for never-heard-of rule
184		request, channel = self.make_request(
185		"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
186		)
187		self.assertEqual(channel.code, 404)
188		self.assertEqual(channel.json_body["errcode"], Codes.NOT_FOUND)
189
190		# PUT a new rule
191		request, channel = self.make_request(
	184	channel = self.make_request(
	185	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
	186	)
	187	self.assertEqual(channel.code, 404)
	188	self.assertEqual(channel.json_body["errcode"], Codes.NOT_FOUND)
	189
	190	# PUT a new rule
	191	channel = self.make_request(
192	192	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
193	193	)
194	194	self.assertEqual(channel.code, 200)
195	195
196	196	# GET enabled for that new rule
197		request, channel = self.make_request(
	197	channel = self.make_request(
198	198	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
199	199	)
200	200	self.assertEqual(channel.code, 200)
201	201
202	202	# DELETE the rule
203		request, channel = self.make_request(
	203	channel = self.make_request(
204	204	"DELETE", "/pushrules/global/override/best.friend", access_token=token
205	205	)
206	206	self.assertEqual(channel.code, 200)
207	207
208	208	# check 404 for deleted rule
209		request, channel = self.make_request(
	209	channel = self.make_request(
210	210	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
211	211	)
212	212	self.assertEqual(channel.code, 404)

220	220	token = self.login("user", "pass")
221	221
222	222	# check 404 for never-heard-of rule
223		request, channel = self.make_request(
	223	channel = self.make_request(
224	224	"GET", "/pushrules/global/override/.m.muahahaha/enabled", access_token=token
225	225	)
226	226	self.assertEqual(channel.code, 404)

234	234	token = self.login("user", "pass")
235	235
236	236	# enable & check 404 for never-heard-of rule
237		request, channel = self.make_request(
	237	channel = self.make_request(
238	238	"PUT",
239	239	"/pushrules/global/override/best.friend/enabled",
240	240	{"enabled": True},

251	251	token = self.login("user", "pass")
252	252
253	253	# enable & check 404 for never-heard-of rule
254		request, channel = self.make_request(
	254	channel = self.make_request(
255	255	"PUT",
256	256	"/pushrules/global/override/.m.muahahah/enabled",
257	257	{"enabled": True},

275	275	}
276	276
277	277	# PUT a new rule
278		request, channel = self.make_request(
	278	channel = self.make_request(
279	279	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
280	280	)
281	281	self.assertEqual(channel.code, 200)
282	282
283	283	# GET actions for that new rule
284		request, channel = self.make_request(
	284	channel = self.make_request(
285	285	"GET", "/pushrules/global/override/best.friend/actions", access_token=token
286	286	)
287	287	self.assertEqual(channel.code, 200)

304	304	}
305	305
306	306	# PUT a new rule
307		request, channel = self.make_request(
	307	channel = self.make_request(
308	308	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
309	309	)
310	310	self.assertEqual(channel.code, 200)
311	311
312	312	# change the rule actions
313		request, channel = self.make_request(
	313	channel = self.make_request(
314	314	"PUT",
315	315	"/pushrules/global/override/best.friend/actions",
316	316	{"actions": ["dont_notify"]},

319	319	self.assertEqual(channel.code, 200)
320	320
321	321	# GET actions for that new rule
322		request, channel = self.make_request(
	322	channel = self.make_request(
323	323	"GET", "/pushrules/global/override/best.friend/actions", access_token=token
324	324	)
325	325	self.assertEqual(channel.code, 200)

340	340	}
341	341
342	342	# check 404 for never-heard-of rule
343		request, channel = self.make_request(
344		"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
345		)
346		self.assertEqual(channel.code, 404)
347		self.assertEqual(channel.json_body["errcode"], Codes.NOT_FOUND)
348
349		# PUT a new rule
350		request, channel = self.make_request(
	343	channel = self.make_request(
	344	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
	345	)
	346	self.assertEqual(channel.code, 404)
	347	self.assertEqual(channel.json_body["errcode"], Codes.NOT_FOUND)
	348
	349	# PUT a new rule
	350	channel = self.make_request(
351	351	"PUT", "/pushrules/global/override/best.friend", body, access_token=token
352	352	)
353	353	self.assertEqual(channel.code, 200)
354	354
355	355	# DELETE the rule
356		request, channel = self.make_request(
	356	channel = self.make_request(
357	357	"DELETE", "/pushrules/global/override/best.friend", access_token=token
358	358	)
359	359	self.assertEqual(channel.code, 200)
360	360
361	361	# check 404 for deleted rule
362		request, channel = self.make_request(
	362	channel = self.make_request(
363	363	"GET", "/pushrules/global/override/best.friend/enabled", access_token=token
364	364	)
365	365	self.assertEqual(channel.code, 404)

373	373	token = self.login("user", "pass")
374	374
375	375	# check 404 for never-heard-of rule
376		request, channel = self.make_request(
	376	channel = self.make_request(
377	377	"GET", "/pushrules/global/override/.m.muahahaha/actions", access_token=token
378	378	)
379	379	self.assertEqual(channel.code, 404)

387	387	token = self.login("user", "pass")
388	388
389	389	# enable & check 404 for never-heard-of rule
390		request, channel = self.make_request(
	390	channel = self.make_request(
391	391	"PUT",
392	392	"/pushrules/global/override/best.friend/actions",
393	393	{"actions": ["dont_notify"]},

404	404	token = self.login("user", "pass")
405	405
406	406	# enable & check 404 for never-heard-of rule
407		request, channel = self.make_request(
	407	channel = self.make_request(
408	408	"PUT",
409	409	"/pushrules/global/override/.m.muahahah/actions",
410	410	{"actions": ["dont_notify"]},

+152

-146

tests/rest/client/v1/test_rooms.py less more

25	25	import synapse.rest.admin
26	26	from synapse.api.constants import EventContentFields, EventTypes, Membership
27	27	from synapse.handlers.pagination import PurgeStatus
	28	from synapse.rest import admin
28	29	from synapse.rest.client.v1 import directory, login, profile, room
29	30	from synapse.rest.client.v2_alpha import account
30	31	from synapse.types import JsonDict, RoomAlias, UserID

44	45	def make_homeserver(self, reactor, clock):
45	46
46	47	self.hs = self.setup_test_homeserver(
47		"red", http_client=None, federation_client=Mock(),
	48	"red", federation_http_client=None, federation_client=Mock(),
48	49	)
49	50
50	51	self.hs.get_federation_handler = Mock()

82	83	self.created_rmid_msg_path = (
83	84	"rooms/%s/send/m.room.message/a1" % (self.created_rmid)
84	85	).encode("ascii")
85		request, channel = self.make_request(
	86	channel = self.make_request(
86	87	"PUT", self.created_rmid_msg_path, b'{"msgtype":"m.text","body":"test msg"}'
87	88	)
88	89	self.assertEquals(200, channel.code, channel.result)
89	90
90	91	# set topic for public room
91		request, channel = self.make_request(
	92	channel = self.make_request(
92	93	"PUT",
93	94	("rooms/%s/state/m.room.topic" % self.created_public_rmid).encode("ascii"),
94	95	b'{"topic":"Public Room Topic"}',

110	111	)
111	112
112	113	# send message in uncreated room, expect 403
113		request, channel = self.make_request(
	114	channel = self.make_request(
114	115	"PUT",
115	116	"/rooms/%s/send/m.room.message/mid2" % (self.uncreated_rmid,),
116	117	msg_content,

118	119	self.assertEquals(403, channel.code, msg=channel.result["body"])
119	120
120	121	# send message in created room not joined (no state), expect 403
121		request, channel = self.make_request("PUT", send_msg_path(), msg_content)
	122	channel = self.make_request("PUT", send_msg_path(), msg_content)
122	123	self.assertEquals(403, channel.code, msg=channel.result["body"])
123	124
124	125	# send message in created room and invited, expect 403
125	126	self.helper.invite(
126	127	room=self.created_rmid, src=self.rmcreator_id, targ=self.user_id
127	128	)
128		request, channel = self.make_request("PUT", send_msg_path(), msg_content)
	129	channel = self.make_request("PUT", send_msg_path(), msg_content)
129	130	self.assertEquals(403, channel.code, msg=channel.result["body"])
130	131
131	132	# send message in created room and joined, expect 200
132	133	self.helper.join(room=self.created_rmid, user=self.user_id)
133		request, channel = self.make_request("PUT", send_msg_path(), msg_content)
	134	channel = self.make_request("PUT", send_msg_path(), msg_content)
134	135	self.assertEquals(200, channel.code, msg=channel.result["body"])
135	136
136	137	# send message in created room and left, expect 403
137	138	self.helper.leave(room=self.created_rmid, user=self.user_id)
138		request, channel = self.make_request("PUT", send_msg_path(), msg_content)
	139	channel = self.make_request("PUT", send_msg_path(), msg_content)
139	140	self.assertEquals(403, channel.code, msg=channel.result["body"])
140	141
141	142	def test_topic_perms(self):

143	144	topic_path = "/rooms/%s/state/m.room.topic" % self.created_rmid
144	145
145	146	# set/get topic in uncreated room, expect 403
146		request, channel = self.make_request(
	147	channel = self.make_request(
147	148	"PUT", "/rooms/%s/state/m.room.topic" % self.uncreated_rmid, topic_content
148	149	)
149	150	self.assertEquals(403, channel.code, msg=channel.result["body"])
150		request, channel = self.make_request(
	151	channel = self.make_request(
151	152	"GET", "/rooms/%s/state/m.room.topic" % self.uncreated_rmid
152	153	)
153	154	self.assertEquals(403, channel.code, msg=channel.result["body"])
154	155
155	156	# set/get topic in created PRIVATE room not joined, expect 403
156		request, channel = self.make_request("PUT", topic_path, topic_content)
	157	channel = self.make_request("PUT", topic_path, topic_content)
157	158	self.assertEquals(403, channel.code, msg=channel.result["body"])
158		request, channel = self.make_request("GET", topic_path)
	159	channel = self.make_request("GET", topic_path)
159	160	self.assertEquals(403, channel.code, msg=channel.result["body"])
160	161
161	162	# set topic in created PRIVATE room and invited, expect 403
162	163	self.helper.invite(
163	164	room=self.created_rmid, src=self.rmcreator_id, targ=self.user_id
164	165	)
165		request, channel = self.make_request("PUT", topic_path, topic_content)
	166	channel = self.make_request("PUT", topic_path, topic_content)
166	167	self.assertEquals(403, channel.code, msg=channel.result["body"])
167	168
168	169	# get topic in created PRIVATE room and invited, expect 403
169		request, channel = self.make_request("GET", topic_path)
	170	channel = self.make_request("GET", topic_path)
170	171	self.assertEquals(403, channel.code, msg=channel.result["body"])
171	172
172	173	# set/get topic in created PRIVATE room and joined, expect 200

174	175
175	176	# Only room ops can set topic by default
176	177	self.helper.auth_user_id = self.rmcreator_id
177		request, channel = self.make_request("PUT", topic_path, topic_content)
	178	channel = self.make_request("PUT", topic_path, topic_content)
178	179	self.assertEquals(200, channel.code, msg=channel.result["body"])
179	180	self.helper.auth_user_id = self.user_id
180	181
181		request, channel = self.make_request("GET", topic_path)
	182	channel = self.make_request("GET", topic_path)
182	183	self.assertEquals(200, channel.code, msg=channel.result["body"])
183	184	self.assert_dict(json.loads(topic_content.decode("utf8")), channel.json_body)
184	185
185	186	# set/get topic in created PRIVATE room and left, expect 403
186	187	self.helper.leave(room=self.created_rmid, user=self.user_id)
187		request, channel = self.make_request("PUT", topic_path, topic_content)
	188	channel = self.make_request("PUT", topic_path, topic_content)
188	189	self.assertEquals(403, channel.code, msg=channel.result["body"])
189		request, channel = self.make_request("GET", topic_path)
	190	channel = self.make_request("GET", topic_path)
190	191	self.assertEquals(200, channel.code, msg=channel.result["body"])
191	192
192	193	# get topic in PUBLIC room, not joined, expect 403
193		request, channel = self.make_request(
	194	channel = self.make_request(
194	195	"GET", "/rooms/%s/state/m.room.topic" % self.created_public_rmid
195	196	)
196	197	self.assertEquals(403, channel.code, msg=channel.result["body"])
197	198
198	199	# set topic in PUBLIC room, not joined, expect 403
199		request, channel = self.make_request(
	200	channel = self.make_request(
200	201	"PUT",
201	202	"/rooms/%s/state/m.room.topic" % self.created_public_rmid,
202	203	topic_content,

206	207	def _test_get_membership(self, room=None, members=[], expect_code=None):
207	208	for member in members:
208	209	path = "/rooms/%s/state/m.room.member/%s" % (room, member)
209		request, channel = self.make_request("GET", path)
	210	channel = self.make_request("GET", path)
210	211	self.assertEquals(expect_code, channel.code)
211	212
212	213	def test_membership_basic_room_perms(self):

378	379
379	380	def test_get_member_list(self):
380	381	room_id = self.helper.create_room_as(self.user_id)
381		request, channel = self.make_request("GET", "/rooms/%s/members" % room_id)
	382	channel = self.make_request("GET", "/rooms/%s/members" % room_id)
382	383	self.assertEquals(200, channel.code, msg=channel.result["body"])
383	384
384	385	def test_get_member_list_no_room(self):
385		request, channel = self.make_request("GET", "/rooms/roomdoesnotexist/members")
	386	channel = self.make_request("GET", "/rooms/roomdoesnotexist/members")
386	387	self.assertEquals(403, channel.code, msg=channel.result["body"])
387	388
388	389	def test_get_member_list_no_permission(self):
389	390	room_id = self.helper.create_room_as("@some_other_guy:red")
390		request, channel = self.make_request("GET", "/rooms/%s/members" % room_id)
	391	channel = self.make_request("GET", "/rooms/%s/members" % room_id)
391	392	self.assertEquals(403, channel.code, msg=channel.result["body"])
392	393
393	394	def test_get_member_list_mixed_memberships(self):

396	397	room_path = "/rooms/%s/members" % room_id
397	398	self.helper.invite(room=room_id, src=room_creator, targ=self.user_id)
398	399	# can't see list if you're just invited.
399		request, channel = self.make_request("GET", room_path)
	400	channel = self.make_request("GET", room_path)
400	401	self.assertEquals(403, channel.code, msg=channel.result["body"])
401	402
402	403	self.helper.join(room=room_id, user=self.user_id)
403	404	# can see list now joined
404		request, channel = self.make_request("GET", room_path)
	405	channel = self.make_request("GET", room_path)
405	406	self.assertEquals(200, channel.code, msg=channel.result["body"])
406	407
407	408	self.helper.leave(room=room_id, user=self.user_id)
408	409	# can see old list once left
409		request, channel = self.make_request("GET", room_path)
	410	channel = self.make_request("GET", room_path)
410	411	self.assertEquals(200, channel.code, msg=channel.result["body"])
411	412
412	413

417	418
418	419	def test_post_room_no_keys(self):
419	420	# POST with no config keys, expect new room id
420		request, channel = self.make_request("POST", "/createRoom", "{}")
	421	channel = self.make_request("POST", "/createRoom", "{}")
421	422
422	423	self.assertEquals(200, channel.code, channel.result)
423	424	self.assertTrue("room_id" in channel.json_body)
424	425
425	426	def test_post_room_visibility_key(self):
426	427	# POST with visibility config key, expect new room id
427		request, channel = self.make_request(
428		"POST", "/createRoom", b'{"visibility":"private"}'
429		)
	428	channel = self.make_request("POST", "/createRoom", b'{"visibility":"private"}')
430	429	self.assertEquals(200, channel.code)
431	430	self.assertTrue("room_id" in channel.json_body)
432	431
433	432	def test_post_room_custom_key(self):
434	433	# POST with custom config keys, expect new room id
435		request, channel = self.make_request(
436		"POST", "/createRoom", b'{"custom":"stuff"}'
437		)
	434	channel = self.make_request("POST", "/createRoom", b'{"custom":"stuff"}')
438	435	self.assertEquals(200, channel.code)
439	436	self.assertTrue("room_id" in channel.json_body)
440	437
441	438	def test_post_room_known_and_unknown_keys(self):
442	439	# POST with custom + known config keys, expect new room id
443		request, channel = self.make_request(
	440	channel = self.make_request(
444	441	"POST", "/createRoom", b'{"visibility":"private","custom":"things"}'
445	442	)
446	443	self.assertEquals(200, channel.code)

448	445
449	446	def test_post_room_invalid_content(self):
450	447	# POST with invalid content / paths, expect 400
451		request, channel = self.make_request("POST", "/createRoom", b'{"visibili')
	448	channel = self.make_request("POST", "/createRoom", b'{"visibili')
452	449	self.assertEquals(400, channel.code)
453	450
454		request, channel = self.make_request("POST", "/createRoom", b'["hello"]')
	451	channel = self.make_request("POST", "/createRoom", b'["hello"]')
455	452	self.assertEquals(400, channel.code)
456	453
457	454	def test_post_room_invitees_invalid_mxid(self):
458	455	# POST with invalid invitee, see https://github.com/matrix-org/synapse/issues/4088
459	456	# Note the trailing space in the MXID here!
460		request, channel = self.make_request(
	457	channel = self.make_request(
461	458	"POST", "/createRoom", b'{"invite":["@alice:example.com "]}'
462	459	)
463	460	self.assertEquals(400, channel.code)

475	472
476	473	def test_invalid_puts(self):
477	474	# missing keys or invalid json
478		request, channel = self.make_request("PUT", self.path, "{}")
479		self.assertEquals(400, channel.code, msg=channel.result["body"])
480
481		request, channel = self.make_request("PUT", self.path, '{"_name":"bo"}')
482		self.assertEquals(400, channel.code, msg=channel.result["body"])
483
484		request, channel = self.make_request("PUT", self.path, '{"nao')
485		self.assertEquals(400, channel.code, msg=channel.result["body"])
486
487		request, channel = self.make_request(
	475	channel = self.make_request("PUT", self.path, "{}")
	476	self.assertEquals(400, channel.code, msg=channel.result["body"])
	477
	478	channel = self.make_request("PUT", self.path, '{"_name":"bo"}')
	479	self.assertEquals(400, channel.code, msg=channel.result["body"])
	480
	481	channel = self.make_request("PUT", self.path, '{"nao')
	482	self.assertEquals(400, channel.code, msg=channel.result["body"])
	483
	484	channel = self.make_request(
488	485	"PUT", self.path, '[{"_name":"bo"},{"_name":"jill"}]'
489	486	)
490	487	self.assertEquals(400, channel.code, msg=channel.result["body"])
491	488
492		request, channel = self.make_request("PUT", self.path, "text only")
493		self.assertEquals(400, channel.code, msg=channel.result["body"])
494
495		request, channel = self.make_request("PUT", self.path, "")
	489	channel = self.make_request("PUT", self.path, "text only")
	490	self.assertEquals(400, channel.code, msg=channel.result["body"])
	491
	492	channel = self.make_request("PUT", self.path, "")
496	493	self.assertEquals(400, channel.code, msg=channel.result["body"])
497	494
498	495	# valid key, wrong type
499	496	content = '{"topic":["Topic name"]}'
500		request, channel = self.make_request("PUT", self.path, content)
	497	channel = self.make_request("PUT", self.path, content)
501	498	self.assertEquals(400, channel.code, msg=channel.result["body"])
502	499
503	500	def test_rooms_topic(self):
504	501	# nothing should be there
505		request, channel = self.make_request("GET", self.path)
	502	channel = self.make_request("GET", self.path)
506	503	self.assertEquals(404, channel.code, msg=channel.result["body"])
507	504
508	505	# valid put
509	506	content = '{"topic":"Topic name"}'
510		request, channel = self.make_request("PUT", self.path, content)
	507	channel = self.make_request("PUT", self.path, content)
511	508	self.assertEquals(200, channel.code, msg=channel.result["body"])
512	509
513	510	# valid get
514		request, channel = self.make_request("GET", self.path)
	511	channel = self.make_request("GET", self.path)
515	512	self.assertEquals(200, channel.code, msg=channel.result["body"])
516	513	self.assert_dict(json.loads(content), channel.json_body)
517	514
518	515	def test_rooms_topic_with_extra_keys(self):
519	516	# valid put with extra keys
520	517	content = '{"topic":"Seasons","subtopic":"Summer"}'
521		request, channel = self.make_request("PUT", self.path, content)
	518	channel = self.make_request("PUT", self.path, content)
522	519	self.assertEquals(200, channel.code, msg=channel.result["body"])
523	520
524	521	# valid get
525		request, channel = self.make_request("GET", self.path)
	522	channel = self.make_request("GET", self.path)
526	523	self.assertEquals(200, channel.code, msg=channel.result["body"])
527	524	self.assert_dict(json.loads(content), channel.json_body)
528	525

538	535	def test_invalid_puts(self):
539	536	path = "/rooms/%s/state/m.room.member/%s" % (self.room_id, self.user_id)
540	537	# missing keys or invalid json
541		request, channel = self.make_request("PUT", path, "{}")
542		self.assertEquals(400, channel.code, msg=channel.result["body"])
543
544		request, channel = self.make_request("PUT", path, '{"_name":"bo"}')
545		self.assertEquals(400, channel.code, msg=channel.result["body"])
546
547		request, channel = self.make_request("PUT", path, '{"nao')
548		self.assertEquals(400, channel.code, msg=channel.result["body"])
549
550		request, channel = self.make_request(
551		"PUT", path, b'[{"_name":"bo"},{"_name":"jill"}]'
552		)
553		self.assertEquals(400, channel.code, msg=channel.result["body"])
554
555		request, channel = self.make_request("PUT", path, "text only")
556		self.assertEquals(400, channel.code, msg=channel.result["body"])
557
558		request, channel = self.make_request("PUT", path, "")
	538	channel = self.make_request("PUT", path, "{}")
	539	self.assertEquals(400, channel.code, msg=channel.result["body"])
	540
	541	channel = self.make_request("PUT", path, '{"_name":"bo"}')
	542	self.assertEquals(400, channel.code, msg=channel.result["body"])
	543
	544	channel = self.make_request("PUT", path, '{"nao')
	545	self.assertEquals(400, channel.code, msg=channel.result["body"])
	546
	547	channel = self.make_request("PUT", path, b'[{"_name":"bo"},{"_name":"jill"}]')
	548	self.assertEquals(400, channel.code, msg=channel.result["body"])
	549
	550	channel = self.make_request("PUT", path, "text only")
	551	self.assertEquals(400, channel.code, msg=channel.result["body"])
	552
	553	channel = self.make_request("PUT", path, "")
559	554	self.assertEquals(400, channel.code, msg=channel.result["body"])
560	555
561	556	# valid keys, wrong types

564	559	Membership.JOIN,
565	560	Membership.LEAVE,
566	561	)
567		request, channel = self.make_request("PUT", path, content.encode("ascii"))
	562	channel = self.make_request("PUT", path, content.encode("ascii"))
568	563	self.assertEquals(400, channel.code, msg=channel.result["body"])
569	564
570	565	def test_rooms_members_self(self):

575	570
576	571	# valid join message (NOOP since we made the room)
577	572	content = '{"membership":"%s"}' % Membership.JOIN
578		request, channel = self.make_request("PUT", path, content.encode("ascii"))
	573	channel = self.make_request("PUT", path, content.encode("ascii"))
579	574	self.assertEquals(200, channel.code, msg=channel.result["body"])
580	575
581		request, channel = self.make_request("GET", path, None)
	576	channel = self.make_request("GET", path, None)
582	577	self.assertEquals(200, channel.code, msg=channel.result["body"])
583	578
584	579	expected_response = {"membership": Membership.JOIN}

593	588
594	589	# valid invite message
595	590	content = '{"membership":"%s"}' % Membership.INVITE
596		request, channel = self.make_request("PUT", path, content)
	591	channel = self.make_request("PUT", path, content)
597	592	self.assertEquals(200, channel.code, msg=channel.result["body"])
598	593
599		request, channel = self.make_request("GET", path, None)
	594	channel = self.make_request("GET", path, None)
600	595	self.assertEquals(200, channel.code, msg=channel.result["body"])
601	596	self.assertEquals(json.loads(content), channel.json_body)
602	597

612	607	Membership.INVITE,
613	608	"Join us!",
614	609	)
615		request, channel = self.make_request("PUT", path, content)
	610	channel = self.make_request("PUT", path, content)
616	611	self.assertEquals(200, channel.code, msg=channel.result["body"])
617	612
618		request, channel = self.make_request("GET", path, None)
	613	channel = self.make_request("GET", path, None)
619	614	self.assertEquals(200, channel.code, msg=channel.result["body"])
620	615	self.assertEquals(json.loads(content), channel.json_body)
621	616

624	619	user_id = "@sid1:red"
625	620
626	621	servlets = [
	622	admin.register_servlets,
627	623	profile.register_servlets,
628	624	room.register_servlets,
629	625	]

665	661
666	662	# Update the display name for the user.
667	663	path = "/_matrix/client/r0/profile/%s/displayname" % self.user_id
668		request, channel = self.make_request("PUT", path, {"displayname": "John Doe"})
	664	channel = self.make_request("PUT", path, {"displayname": "John Doe"})
669	665	self.assertEquals(channel.code, 200, channel.json_body)
670	666
671	667	# Check that all the rooms have been sent a profile update into.

675	671	self.user_id,
676	672	)
677	673
678		request, channel = self.make_request("GET", path)
	674	channel = self.make_request("GET", path)
679	675	self.assertEquals(channel.code, 200)
680	676
681	677	self.assertIn("displayname", channel.json_body)

699	695	# Make sure we send more requests than the rate-limiting config would allow
700	696	# if all of these requests ended up joining the user to a room.
701	697	for i in range(4):
702		request, channel = self.make_request("POST", path % room_id, {})
	698	channel = self.make_request("POST", path % room_id, {})
703	699	self.assertEquals(channel.code, 200)
	700
	701	@unittest.override_config(
	702	{
	703	"rc_joins": {"local": {"per_second": 0.5, "burst_count": 3}},
	704	"auto_join_rooms": ["#room:red", "#room2:red", "#room3:red", "#room4:red"],
	705	"autocreate_auto_join_rooms": True,
	706	},
	707	)
	708	def test_autojoin_rooms(self):
	709	user_id = self.register_user("testuser", "password")
	710
	711	# Check that the new user successfully joined the four rooms
	712	rooms = self.get_success(self.hs.get_datastore().get_rooms_for_user(user_id))
	713	self.assertEqual(len(rooms), 4)
704	714
705	715
706	716	class RoomMessagesTestCase(RoomBase):

714	724	def test_invalid_puts(self):
715	725	path = "/rooms/%s/send/m.room.message/mid1" % (urlparse.quote(self.room_id))
716	726	# missing keys or invalid json
717		request, channel = self.make_request("PUT", path, b"{}")
718		self.assertEquals(400, channel.code, msg=channel.result["body"])
719
720		request, channel = self.make_request("PUT", path, b'{"_name":"bo"}')
721		self.assertEquals(400, channel.code, msg=channel.result["body"])
722
723		request, channel = self.make_request("PUT", path, b'{"nao')
724		self.assertEquals(400, channel.code, msg=channel.result["body"])
725
726		request, channel = self.make_request(
727		"PUT", path, b'[{"_name":"bo"},{"_name":"jill"}]'
728		)
729		self.assertEquals(400, channel.code, msg=channel.result["body"])
730
731		request, channel = self.make_request("PUT", path, b"text only")
732		self.assertEquals(400, channel.code, msg=channel.result["body"])
733
734		request, channel = self.make_request("PUT", path, b"")
	727	channel = self.make_request("PUT", path, b"{}")
	728	self.assertEquals(400, channel.code, msg=channel.result["body"])
	729
	730	channel = self.make_request("PUT", path, b'{"_name":"bo"}')
	731	self.assertEquals(400, channel.code, msg=channel.result["body"])
	732
	733	channel = self.make_request("PUT", path, b'{"nao')
	734	self.assertEquals(400, channel.code, msg=channel.result["body"])
	735
	736	channel = self.make_request("PUT", path, b'[{"_name":"bo"},{"_name":"jill"}]')
	737	self.assertEquals(400, channel.code, msg=channel.result["body"])
	738
	739	channel = self.make_request("PUT", path, b"text only")
	740	self.assertEquals(400, channel.code, msg=channel.result["body"])
	741
	742	channel = self.make_request("PUT", path, b"")
735	743	self.assertEquals(400, channel.code, msg=channel.result["body"])
736	744
737	745	def test_rooms_messages_sent(self):
738	746	path = "/rooms/%s/send/m.room.message/mid1" % (urlparse.quote(self.room_id))
739	747
740	748	content = b'{"body":"test","msgtype":{"type":"a"}}'
741		request, channel = self.make_request("PUT", path, content)
	749	channel = self.make_request("PUT", path, content)
742	750	self.assertEquals(400, channel.code, msg=channel.result["body"])
743	751
744	752	# custom message types
745	753	content = b'{"body":"test","msgtype":"test.custom.text"}'
746		request, channel = self.make_request("PUT", path, content)
	754	channel = self.make_request("PUT", path, content)
747	755	self.assertEquals(200, channel.code, msg=channel.result["body"])
748	756
749	757	# m.text message type
750	758	path = "/rooms/%s/send/m.room.message/mid2" % (urlparse.quote(self.room_id))
751	759	content = b'{"body":"test2","msgtype":"m.text"}'
752		request, channel = self.make_request("PUT", path, content)
	760	channel = self.make_request("PUT", path, content)
753	761	self.assertEquals(200, channel.code, msg=channel.result["body"])
754	762
755	763

763	771	self.room_id = self.helper.create_room_as(self.user_id)
764	772
765	773	def test_initial_sync(self):
766		request, channel = self.make_request(
767		"GET", "/rooms/%s/initialSync" % self.room_id
768		)
	774	channel = self.make_request("GET", "/rooms/%s/initialSync" % self.room_id)
769	775	self.assertEquals(200, channel.code)
770	776
771	777	self.assertEquals(self.room_id, channel.json_body["room_id"])

806	812
807	813	def test_topo_token_is_accepted(self):
808	814	token = "t1-0_0_0_0_0_0_0_0_0"
809		request, channel = self.make_request(
	815	channel = self.make_request(
810	816	"GET", "/rooms/%s/messages?access_token=x&from=%s" % (self.room_id, token)
811	817	)
812	818	self.assertEquals(200, channel.code)

817	823
818	824	def test_stream_token_is_accepted_for_fwd_pagianation(self):
819	825	token = "s0_0_0_0_0_0_0_0_0"
820		request, channel = self.make_request(
	826	channel = self.make_request(
821	827	"GET", "/rooms/%s/messages?access_token=x&from=%s" % (self.room_id, token)
822	828	)
823	829	self.assertEquals(200, channel.code)

850	856	self.helper.send(self.room_id, "message 3")
851	857
852	858	# Check that we get the first and second message when querying /messages.
853		request, channel = self.make_request(
	859	channel = self.make_request(
854	860	"GET",
855	861	"/rooms/%s/messages?access_token=x&from=%s&dir=b&filter=%s"
856	862	% (

878	884
879	885	# Check that we only get the second message through /message now that the first
880	886	# has been purged.
881		request, channel = self.make_request(
	887	channel = self.make_request(
882	888	"GET",
883	889	"/rooms/%s/messages?access_token=x&from=%s&dir=b&filter=%s"
884	890	% (

895	901	# Check that we get no event, but also no error, when querying /messages with
896	902	# the token that was pointing at the first event, because we don't have it
897	903	# anymore.
898		request, channel = self.make_request(
	904	channel = self.make_request(
899	905	"GET",
900	906	"/rooms/%s/messages?access_token=x&from=%s&dir=b&filter=%s"
901	907	% (

954	960	self.helper.send(self.room, body="Hi!", tok=self.other_access_token)
955	961	self.helper.send(self.room, body="There!", tok=self.other_access_token)
956	962
957		request, channel = self.make_request(
	963	channel = self.make_request(
958	964	"POST",
959	965	"/search?access_token=%s" % (self.access_token,),
960	966	{

983	989	self.helper.send(self.room, body="Hi!", tok=self.other_access_token)
984	990	self.helper.send(self.room, body="There!", tok=self.other_access_token)
985	991
986		request, channel = self.make_request(
	992	channel = self.make_request(
987	993	"POST",
988	994	"/search?access_token=%s" % (self.access_token,),
989	995	{

1031	1037	return self.hs
1032	1038
1033	1039	def test_restricted_no_auth(self):
1034		request, channel = self.make_request("GET", self.url)
	1040	channel = self.make_request("GET", self.url)
1035	1041	self.assertEqual(channel.code, 401, channel.result)
1036	1042
1037	1043	def test_restricted_auth(self):
1038	1044	self.register_user("user", "pass")
1039	1045	tok = self.login("user", "pass")
1040	1046
1041		request, channel = self.make_request("GET", self.url, access_token=tok)
	1047	channel = self.make_request("GET", self.url, access_token=tok)
1042	1048	self.assertEqual(channel.code, 200, channel.result)
1043	1049
1044	1050

1066	1072	self.displayname = "test user"
1067	1073	data = {"displayname": self.displayname}
1068	1074	request_data = json.dumps(data)
1069		request, channel = self.make_request(
	1075	channel = self.make_request(
1070	1076	"PUT",
1071	1077	"/_matrix/client/r0/profile/%s/displayname" % (self.user_id,),
1072	1078	request_data,

1079	1085	def test_per_room_profile_forbidden(self):
1080	1086	data = {"membership": "join", "displayname": "other test user"}
1081	1087	request_data = json.dumps(data)
1082		request, channel = self.make_request(
	1088	channel = self.make_request(
1083	1089	"PUT",
1084	1090	"/_matrix/client/r0/rooms/%s/state/m.room.member/%s"
1085	1091	% (self.room_id, self.user_id),

1089	1095	self.assertEqual(channel.code, 200, channel.result)
1090	1096	event_id = channel.json_body["event_id"]
1091	1097
1092		request, channel = self.make_request(
	1098	channel = self.make_request(
1093	1099	"GET",
1094	1100	"/_matrix/client/r0/rooms/%s/event/%s" % (self.room_id, event_id),
1095	1101	access_token=self.tok,

1122	1128
1123	1129	def test_join_reason(self):
1124	1130	reason = "hello"
1125		request, channel = self.make_request(
	1131	channel = self.make_request(
1126	1132	"POST",
1127	1133	"/_matrix/client/r0/rooms/{}/join".format(self.room_id),
1128	1134	content={"reason": reason},

1136	1142	self.helper.join(self.room_id, user=self.second_user_id, tok=self.second_tok)
1137	1143
1138	1144	reason = "hello"
1139		request, channel = self.make_request(
	1145	channel = self.make_request(
1140	1146	"POST",
1141	1147	"/_matrix/client/r0/rooms/{}/leave".format(self.room_id),
1142	1148	content={"reason": reason},

1150	1156	self.helper.join(self.room_id, user=self.second_user_id, tok=self.second_tok)
1151	1157
1152	1158	reason = "hello"
1153		request, channel = self.make_request(
	1159	channel = self.make_request(
1154	1160	"POST",
1155	1161	"/_matrix/client/r0/rooms/{}/kick".format(self.room_id),
1156	1162	content={"reason": reason, "user_id": self.second_user_id},

1164	1170	self.helper.join(self.room_id, user=self.second_user_id, tok=self.second_tok)
1165	1171
1166	1172	reason = "hello"
1167		request, channel = self.make_request(
	1173	channel = self.make_request(
1168	1174	"POST",
1169	1175	"/_matrix/client/r0/rooms/{}/ban".format(self.room_id),
1170	1176	content={"reason": reason, "user_id": self.second_user_id},

1176	1182
1177	1183	def test_unban_reason(self):
1178	1184	reason = "hello"
1179		request, channel = self.make_request(
	1185	channel = self.make_request(
1180	1186	"POST",
1181	1187	"/_matrix/client/r0/rooms/{}/unban".format(self.room_id),
1182	1188	content={"reason": reason, "user_id": self.second_user_id},

1188	1194
1189	1195	def test_invite_reason(self):
1190	1196	reason = "hello"
1191		request, channel = self.make_request(
	1197	channel = self.make_request(
1192	1198	"POST",
1193	1199	"/_matrix/client/r0/rooms/{}/invite".format(self.room_id),
1194	1200	content={"reason": reason, "user_id": self.second_user_id},

1207	1213	)
1208	1214
1209	1215	reason = "hello"
1210		request, channel = self.make_request(
	1216	channel = self.make_request(
1211	1217	"POST",
1212	1218	"/_matrix/client/r0/rooms/{}/leave".format(self.room_id),
1213	1219	content={"reason": reason},

1218	1224	self._check_for_reason(reason)
1219	1225
1220	1226	def _check_for_reason(self, reason):
1221		request, channel = self.make_request(
	1227	channel = self.make_request(
1222	1228	"GET",
1223	1229	"/_matrix/client/r0/rooms/{}/state/m.room.member/{}".format(
1224	1230	self.room_id, self.second_user_id

1267	1273	"""Test that we can filter by a label on a /context request."""
1268	1274	event_id = self._send_labelled_messages_in_room()
1269	1275
1270		request, channel = self.make_request(
	1276	channel = self.make_request(
1271	1277	"GET",
1272	1278	"/rooms/%s/context/%s?filter=%s"
1273	1279	% (self.room_id, event_id, json.dumps(self.FILTER_LABELS)),

1297	1303	"""Test that we can filter by the absence of a label on a /context request."""
1298	1304	event_id = self._send_labelled_messages_in_room()
1299	1305
1300		request, channel = self.make_request(
	1306	channel = self.make_request(
1301	1307	"GET",
1302	1308	"/rooms/%s/context/%s?filter=%s"
1303	1309	% (self.room_id, event_id, json.dumps(self.FILTER_NOT_LABELS)),

1332	1338	"""
1333	1339	event_id = self._send_labelled_messages_in_room()
1334	1340
1335		request, channel = self.make_request(
	1341	channel = self.make_request(
1336	1342	"GET",
1337	1343	"/rooms/%s/context/%s?filter=%s"
1338	1344	% (self.room_id, event_id, json.dumps(self.FILTER_LABELS_NOT_LABELS)),

1360	1366	self._send_labelled_messages_in_room()
1361	1367
1362	1368	token = "s0_0_0_0_0_0_0_0_0"
1363		request, channel = self.make_request(
	1369	channel = self.make_request(
1364	1370	"GET",
1365	1371	"/rooms/%s/messages?access_token=%s&from=%s&filter=%s"
1366	1372	% (self.room_id, self.tok, token, json.dumps(self.FILTER_LABELS)),

1377	1383	self._send_labelled_messages_in_room()
1378	1384
1379	1385	token = "s0_0_0_0_0_0_0_0_0"
1380		request, channel = self.make_request(
	1386	channel = self.make_request(
1381	1387	"GET",
1382	1388	"/rooms/%s/messages?access_token=%s&from=%s&filter=%s"
1383	1389	% (self.room_id, self.tok, token, json.dumps(self.FILTER_NOT_LABELS)),

1400	1406	self._send_labelled_messages_in_room()
1401	1407
1402	1408	token = "s0_0_0_0_0_0_0_0_0"
1403		request, channel = self.make_request(
	1409	channel = self.make_request(
1404	1410	"GET",
1405	1411	"/rooms/%s/messages?access_token=%s&from=%s&filter=%s"
1406	1412	% (

1431	1437
1432	1438	self._send_labelled_messages_in_room()
1433	1439
1434		request, channel = self.make_request(
	1440	channel = self.make_request(
1435	1441	"POST", "/search?access_token=%s" % self.tok, request_data
1436	1442	)
1437	1443

1466	1472
1467	1473	self._send_labelled_messages_in_room()
1468	1474
1469		request, channel = self.make_request(
	1475	channel = self.make_request(
1470	1476	"POST", "/search?access_token=%s" % self.tok, request_data
1471	1477	)
1472	1478

1513	1519
1514	1520	self._send_labelled_messages_in_room()
1515	1521
1516		request, channel = self.make_request(
	1522	channel = self.make_request(
1517	1523	"POST", "/search?access_token=%s" % self.tok, request_data
1518	1524	)
1519	1525

1634	1640
1635	1641	# Check that we can still see the messages before the erasure request.
1636	1642
1637		request, channel = self.make_request(
	1643	channel = self.make_request(
1638	1644	"GET",
1639	1645	'/rooms/%s/context/%s?filter={"types":["m.room.message"]}'
1640	1646	% (self.room_id, event_id),

1698	1704	# Check that a user that joined the room after the erasure request can't see
1699	1705	# the messages anymore.
1700	1706
1701		request, channel = self.make_request(
	1707	channel = self.make_request(
1702	1708	"GET",
1703	1709	'/rooms/%s/context/%s?filter={"types":["m.room.message"]}'
1704	1710	% (self.room_id, event_id),

1788	1794
1789	1795	def _get_aliases(self, access_token: str, expected_code: int = 200) -> JsonDict:
1790	1796	"""Calls the endpoint under test. returns the json response object."""
1791		request, channel = self.make_request(
	1797	channel = self.make_request(
1792	1798	"GET",
1793	1799	"/_matrix/client/unstable/org.matrix.msc2432/rooms/%s/aliases"
1794	1800	% (self.room_id,),

1809	1815	data = {"room_id": self.room_id}
1810	1816	request_data = json.dumps(data)
1811	1817
1812		request, channel = self.make_request(
	1818	channel = self.make_request(
1813	1819	"PUT", url, request_data, access_token=self.room_owner_tok
1814	1820	)
1815	1821	self.assertEqual(channel.code, expected_code, channel.result)

1839	1845	data = {"room_id": self.room_id}
1840	1846	request_data = json.dumps(data)
1841	1847
1842		request, channel = self.make_request(
	1848	channel = self.make_request(
1843	1849	"PUT", url, request_data, access_token=self.room_owner_tok
1844	1850	)
1845	1851	self.assertEqual(channel.code, expected_code, channel.result)
1846	1852
1847	1853	def _get_canonical_alias(self, expected_code: int = 200) -> JsonDict:
1848	1854	"""Calls the endpoint under test. returns the json response object."""
1849		request, channel = self.make_request(
	1855	channel = self.make_request(
1850	1856	"GET",
1851	1857	"rooms/%s/state/m.room.canonical_alias" % (self.room_id,),
1852	1858	access_token=self.room_owner_tok,

1858	1864
1859	1865	def _set_canonical_alias(self, content: str, expected_code: int = 200) -> JsonDict:
1860	1866	"""Calls the endpoint under test. returns the json response object."""
1861		request, channel = self.make_request(
	1867	channel = self.make_request(
1862	1868	"PUT",
1863	1869	"rooms/%s/state/m.room.canonical_alias" % (self.room_id,),
1864	1870	json.dumps(content),

+5

-5

tests/rest/client/v1/test_typing.py less more

38	38	def make_homeserver(self, reactor, clock):
39	39
40	40	hs = self.setup_test_homeserver(
41		"red", http_client=None, federation_client=Mock(),
	41	"red", federation_http_client=None, federation_client=Mock(),
42	42	)
43	43
44	44	self.event_source = hs.get_event_sources().sources["typing"]

93	93	self.helper.join(self.room_id, user="@jim:red")
94	94
95	95	def test_set_typing(self):
96		request, channel = self.make_request(
	96	channel = self.make_request(
97	97	"PUT",
98	98	"/rooms/%s/typing/%s" % (self.room_id, self.user_id),
99	99	b'{"typing": true, "timeout": 30000}',

116	116	)
117	117
118	118	def test_set_not_typing(self):
119		request, channel = self.make_request(
	119	channel = self.make_request(
120	120	"PUT",
121	121	"/rooms/%s/typing/%s" % (self.room_id, self.user_id),
122	122	b'{"typing": false}',

124	124	self.assertEquals(200, channel.code)
125	125
126	126	def test_typing_timeout(self):
127		request, channel = self.make_request(
	127	channel = self.make_request(
128	128	"PUT",
129	129	"/rooms/%s/typing/%s" % (self.room_id, self.user_id),
130	130	b'{"typing": true, "timeout": 30000}',

137	137
138	138	self.assertEquals(self.event_source.get_current_key(), 2)
139	139
140		request, channel = self.make_request(
	140	channel = self.make_request(
141	141	"PUT",
142	142	"/rooms/%s/typing/%s" % (self.room_id, self.user_id),
143	143	b'{"typing": true, "timeout": 30000}',

+120

-8

tests/rest/client/v1/utils.py less more

1	1	# Copyright 2014-2016 OpenMarket Ltd
2	2	# Copyright 2017 Vector Creations Ltd
3	3	# Copyright 2018-2019 New Vector Ltd
4		# Copyright 2019 The Matrix.org Foundation C.I.C.
	4	# Copyright 2019-2020 The Matrix.org Foundation C.I.C.
5	5	#
6	6	# Licensed under the Apache License, Version 2.0 (the "License");
7	7	# you may not use this file except in compliance with the License.

16	16	# limitations under the License.
17	17
18	18	import json
	19	import re
19	20	import time
	21	import urllib.parse
20	22	from typing import Any, Dict, Optional
	23
	24	from mock import patch
21	25
22	26	import attr
23	27

25	29	from twisted.web.server import Site
26	30
27	31	from synapse.api.constants import Membership
	32	from synapse.types import JsonDict
28	33
29	34	from tests.server import FakeSite, make_request
	35	from tests.test_utils import FakeResponse
30	36
31	37
32	38	@attr.s

74	80	if tok:
75	81	path = path + "?access_token=%s" % tok
76	82
77		_, channel = make_request(
	83	channel = make_request(
78	84	self.hs.get_reactor(),
79	85	self.site,
80	86	"POST",

150	156	data = {"membership": membership}
151	157	data.update(extra_data)
152	158
153		_, channel = make_request(
	159	channel = make_request(
154	160	self.hs.get_reactor(),
155	161	self.site,
156	162	"PUT",

185	191	if tok:
186	192	path = path + "?access_token=%s" % tok
187	193
188		_, channel = make_request(
	194	channel = make_request(
189	195	self.hs.get_reactor(),
190	196	self.site,
191	197	"PUT",

241	247	if body is not None:
242	248	content = json.dumps(body).encode("utf8")
243	249
244		_, channel = make_request(
245		self.hs.get_reactor(), self.site, method, path, content
246		)
	250	channel = make_request(self.hs.get_reactor(), self.site, method, path, content)
247	251
248	252	assert int(channel.result["code"]) == expect_code, (
249	253	"Expected: %d, got: %d, resp: %r"

326	330	"""
327	331	image_length = len(image_data)
328	332	path = "/_matrix/media/r0/upload?filename=%s" % (filename,)
329		_, channel = make_request(
	333	channel = make_request(
330	334	self.hs.get_reactor(),
331	335	FakeSite(resource),
332	336	"POST",

343	347	)
344	348
345	349	return channel.json_body
	350
	351	def login_via_oidc(self, remote_user_id: str) -> JsonDict:
	352	"""Log in (as a new user) via OIDC
	353
	354	Returns the result of the final token login.
	355
	356	Requires that "oidc_config" in the homeserver config be set appropriately
	357	(TEST_OIDC_CONFIG is a suitable example) - and by implication, needs a
	358	"public_base_url".
	359
	360	Also requires the login servlet and the OIDC callback resource to be mounted at
	361	the normal places.
	362	"""
	363	client_redirect_url = "https://x"
	364
	365	# first hit the redirect url (which will issue a cookie and state)
	366	channel = make_request(
	367	self.hs.get_reactor(),
	368	self.site,
	369	"GET",
	370	"/login/sso/redirect?redirectUrl=" + client_redirect_url,
	371	)
	372	# that will redirect to the OIDC IdP, but we skip that and go straight
	373	# back to synapse's OIDC callback resource. However, we do need the "state"
	374	# param that synapse passes to the IdP via query params, and the cookie that
	375	# synapse passes to the client.
	376	assert channel.code == 302
	377	oauth_uri = channel.headers.getRawHeaders("Location")[0]
	378	params = urllib.parse.parse_qs(urllib.parse.urlparse(oauth_uri).query)
	379	redirect_uri = "%s?%s" % (
	380	urllib.parse.urlparse(params["redirect_uri"][0]).path,
	381	urllib.parse.urlencode({"state": params["state"][0], "code": "TEST_CODE"}),
	382	)
	383	cookies = {}
	384	for h in channel.headers.getRawHeaders("Set-Cookie"):
	385	parts = h.split(";")
	386	k, v = parts[0].split("=", maxsplit=1)
	387	cookies[k] = v
	388
	389	# before we hit the callback uri, stub out some methods in the http client so
	390	# that we don't have to handle full HTTPS requests.
	391
	392	# (expected url, json response) pairs, in the order we expect them.
	393	expected_requests = [
	394	# first we get a hit to the token endpoint, which we tell to return
	395	# a dummy OIDC access token
	396	("https://issuer.test/token", {"access_token": "TEST"}),
	397	# and then one to the user_info endpoint, which returns our remote user id.
	398	("https://issuer.test/userinfo", {"sub": remote_user_id}),
	399	]
	400
	401	async def mock_req(method: str, uri: str, data=None, headers=None):
	402	(expected_uri, resp_obj) = expected_requests.pop(0)
	403	assert uri == expected_uri
	404	resp = FakeResponse(
	405	code=200, phrase=b"OK", body=json.dumps(resp_obj).encode("utf-8"),
	406	)
	407	return resp
	408
	409	with patch.object(self.hs.get_proxied_http_client(), "request", mock_req):
	410	# now hit the callback URI with the right params and a made-up code
	411	channel = make_request(
	412	self.hs.get_reactor(),
	413	self.site,
	414	"GET",
	415	redirect_uri,
	416	custom_headers=[
	417	("Cookie", "%s=%s" % (k, v)) for (k, v) in cookies.items()
	418	],
	419	)
	420
	421	# expect a confirmation page
	422	assert channel.code == 200
	423
	424	# fish the matrix login token out of the body of the confirmation page
	425	m = re.search(
	426	'a href="%s.loginToken=([^"])"' % (client_redirect_url,),
	427	channel.result["body"].decode("utf-8"),
	428	)
	429	assert m
	430	login_token = m.group(1)
	431
	432	# finally, submit the matrix login token to the login API, which gives us our
	433	# matrix access token and device id.
	434	channel = make_request(
	435	self.hs.get_reactor(),
	436	self.site,
	437	"POST",
	438	"/login",
	439	content={"type": "m.login.token", "token": login_token},
	440	)
	441	assert channel.code == 200
	442	return channel.json_body
	443
	444
	445	# an 'oidc_config' suitable for login_via_oidc.
	446	TEST_OIDC_CONFIG = {
	447	"enabled": True,
	448	"discover": False,
	449	"issuer": "https://issuer.test",
	450	"client_id": "test-client-id",
	451	"client_secret": "test-client-secret",
	452	"scopes": ["profile"],
	453	"authorization_endpoint": "https://z",
	454	"token_endpoint": "https://issuer.test/token",
	455	"userinfo_endpoint": "https://issuer.test/userinfo",
	456	"user_mapping_provider": {"config": {"localpart_template": "{{ user.sub }}"}},
	457	}

+24

-34

tests/rest/client/v2_alpha/test_account.py less more

18	18	import re
19	19	from email.parser import Parser
20	20	from typing import Optional
21		from urllib.parse import urlencode
22	21
23	22	import pkg_resources
24	23

240	239	self.assertIsNotNone(session_id)
241	240
242	241	def _request_token(self, email, client_secret):
243		request, channel = self.make_request(
	242	channel = self.make_request(
244	243	"POST",
245	244	b"account/password/email/requestToken",
246	245	{"client_secret": client_secret, "email": email, "send_attempt": 1},

254	253	path = link.replace("https://example.com", "")
255	254
256	255	# Load the password reset confirmation page
257		request, channel = make_request(
	256	channel = make_request(
258	257	self.reactor,
259	258	FakeSite(self.submit_token_resource),
260	259	"GET",

267	266	# Now POST to the same endpoint, mimicking the same behaviour as clicking the
268	267	# password reset confirm button
269	268
270		# Send arguments as url-encoded form data, matching the template's behaviour
271		form_args = []
272		for key, value_list in request.args.items():
273		for value in value_list:
274		arg = (key, value)
275		form_args.append(arg)
276
277	269	# Confirm the password reset
278		request, channel = make_request(
	270	channel = make_request(
279	271	self.reactor,
280	272	FakeSite(self.submit_token_resource),
281	273	"POST",
282	274	path,
283		content=urlencode(form_args).encode("utf8"),
	275	content=b"",
284	276	shorthand=False,
285	277	content_is_form=True,
286	278	)

309	301	def _reset_password(
310	302	self, new_password, session_id, client_secret, expected_code=200
311	303	):
312		request, channel = self.make_request(
	304	channel = self.make_request(
313	305	"POST",
314	306	b"account/password",
315	307	{

351	343	self.assertTrue(self.get_success(store.get_user_deactivated_status(user_id)))
352	344
353	345	# Check that this access token has been invalidated.
354		request, channel = self.make_request("GET", "account/whoami")
355		self.assertEqual(request.code, 401)
	346	channel = self.make_request("GET", "account/whoami")
	347	self.assertEqual(channel.code, 401)
356	348
357	349	def test_pending_invites(self):
358	350	"""Tests that deactivating a user rejects every pending invite for them."""

406	398	"erase": False,
407	399	}
408	400	)
409		request, channel = self.make_request(
	401	channel = self.make_request(
410	402	"POST", "account/deactivate", request_data, access_token=tok
411	403	)
412		self.assertEqual(request.code, 200)
	404	self.assertEqual(channel.code, 200)
413	405
414	406
415	407	class ThreepidEmailRestTestCase(unittest.HomeserverTestCase):

529	521
530	522	self._validate_token(link)
531	523
532		request, channel = self.make_request(
	524	channel = self.make_request(
533	525	"POST",
534	526	b"/_matrix/client/unstable/account/3pid/add",
535	527	{

547	539	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
548	540
549	541	# Get user
550		request, channel = self.make_request(
	542	channel = self.make_request(
551	543	"GET", self.url_3pid, access_token=self.user_id_tok,
552	544	)
553	545

568	560	)
569	561	)
570	562
571		request, channel = self.make_request(
	563	channel = self.make_request(
572	564	"POST",
573	565	b"account/3pid/delete",
574	566	{"medium": "email", "address": self.email},

577	569	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
578	570
579	571	# Get user
580		request, channel = self.make_request(
	572	channel = self.make_request(
581	573	"GET", self.url_3pid, access_token=self.user_id_tok,
582	574	)
583	575

600	592	)
601	593	)
602	594
603		request, channel = self.make_request(
	595	channel = self.make_request(
604	596	"POST",
605	597	b"account/3pid/delete",
606	598	{"medium": "email", "address": self.email},

611	603	self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
612	604
613	605	# Get user
614		request, channel = self.make_request(
	606	channel = self.make_request(
615	607	"GET", self.url_3pid, access_token=self.user_id_tok,
616	608	)
617	609

628	620	self.assertEquals(len(self.email_attempts), 1)
629	621
630	622	# Attempt to add email without clicking the link
631		request, channel = self.make_request(
	623	channel = self.make_request(
632	624	"POST",
633	625	b"/_matrix/client/unstable/account/3pid/add",
634	626	{

646	638	self.assertEqual(Codes.THREEPID_AUTH_FAILED, channel.json_body["errcode"])
647	639
648	640	# Get user
649		request, channel = self.make_request(
	641	channel = self.make_request(
650	642	"GET", self.url_3pid, access_token=self.user_id_tok,
651	643	)
652	644

661	653	session_id = "weasle"
662	654
663	655	# Attempt to add email without even requesting an email
664		request, channel = self.make_request(
	656	channel = self.make_request(
665	657	"POST",
666	658	b"/_matrix/client/unstable/account/3pid/add",
667	659	{

679	671	self.assertEqual(Codes.THREEPID_AUTH_FAILED, channel.json_body["errcode"])
680	672
681	673	# Get user
682		request, channel = self.make_request(
	674	channel = self.make_request(
683	675	"GET", self.url_3pid, access_token=self.user_id_tok,
684	676	)
685	677

783	775	if next_link:
784	776	body["next_link"] = next_link
785	777
786		request, channel = self.make_request(
787		"POST", b"account/3pid/email/requestToken", body,
788		)
	778	channel = self.make_request("POST", b"account/3pid/email/requestToken", body,)
789	779	self.assertEquals(expect_code, channel.code, channel.result)
790	780
791	781	return channel.json_body.get("sid")

793	783	def _request_token_invalid_email(
794	784	self, email, expected_errcode, expected_error, client_secret="foobar",
795	785	):
796		request, channel = self.make_request(
	786	channel = self.make_request(
797	787	"POST",
798	788	b"account/3pid/email/requestToken",
799	789	{"client_secret": client_secret, "email": email, "send_attempt": 1},

806	796	# Remove the host
807	797	path = link.replace("https://example.com", "")
808	798
809		request, channel = self.make_request("GET", path, shorthand=False)
	799	channel = self.make_request("GET", path, shorthand=False)
810	800	self.assertEquals(200, channel.code, channel.result)
811	801
812	802	def _get_link_from_email(self):

840	830
841	831	self._validate_token(link)
842	832
843		request, channel = self.make_request(
	833	channel = self.make_request(
844	834	"POST",
845	835	b"/_matrix/client/unstable/account/3pid/add",
846	836	{

858	848	self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
859	849
860	850	# Get user
861		request, channel = self.make_request(
	851	channel = self.make_request(
862	852	"GET", self.url_3pid, access_token=self.user_id_tok,
863	853	)
864	854

+141

-52

tests/rest/client/v2_alpha/test_auth.py less more

11	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12	12	# See the License for the specific language governing permissions and
13	13	# limitations under the License.
14		from typing import List, Union
	14
	15	from typing import Union
15	16
16	17	from twisted.internet.defer import succeed
17	18
18	19	import synapse.rest.admin
19	20	from synapse.api.constants import LoginType
20	21	from synapse.handlers.ui_auth.checkers import UserInteractiveAuthChecker
21		from synapse.http.site import SynapseRequest
22	22	from synapse.rest.client.v1 import login
23	23	from synapse.rest.client.v2_alpha import auth, devices, register
24		from synapse.types import JsonDict
	24	from synapse.rest.oidc import OIDCResource
	25	from synapse.types import JsonDict, UserID
25	26
26	27	from tests import unittest
	28	from tests.rest.client.v1.utils import TEST_OIDC_CONFIG
27	29	from tests.server import FakeChannel
28	30
29	31

63	65
64	66	def register(self, expected_response: int, body: JsonDict) -> FakeChannel:
65	67	"""Make a register request."""
66		request, channel = self.make_request(
67		"POST", "register", body
68		) # type: SynapseRequest, FakeChannel
69
70		self.assertEqual(request.code, expected_response)
	68	channel = self.make_request("POST", "register", body)
	69
	70	self.assertEqual(channel.code, expected_response)
71	71	return channel
72	72
73	73	def recaptcha(

77	77	if post_session is None:
78	78	post_session = session
79	79
80		request, channel = self.make_request(
	80	channel = self.make_request(
81	81	"GET", "auth/m.login.recaptcha/fallback/web?session=" + session
82		) # type: SynapseRequest, FakeChannel
83		self.assertEqual(request.code, 200)
84
85		request, channel = self.make_request(
	82	)
	83	self.assertEqual(channel.code, 200)
	84
	85	channel = self.make_request(
86	86	"POST",
87	87	"auth/m.login.recaptcha/fallback/web?session="
88	88	+ post_session
89	89	+ "&g-recaptcha-response=a",
90	90	)
91		self.assertEqual(request.code, expected_post_response)
	91	self.assertEqual(channel.code, expected_post_response)
92	92
93	93	# The recaptcha handler is called with the response given
94	94	attempts = self.recaptcha_checker.recaptcha_attempts

155	155	register.register_servlets,
156	156	]
157	157
	158	def default_config(self):
	159	config = super().default_config()
	160
	161	# we enable OIDC as a way of testing SSO flows
	162	oidc_config = {}
	163	oidc_config.update(TEST_OIDC_CONFIG)
	164	oidc_config["allow_existing_users"] = True
	165
	166	config["oidc_config"] = oidc_config
	167	config["public_baseurl"] = "https://synapse.test"
	168	return config
	169
	170	def create_resource_dict(self):
	171	resource_dict = super().create_resource_dict()
	172	# mount the OIDC resource at /_synapse/oidc
	173	resource_dict["/_synapse/oidc"] = OIDCResource(self.hs)
	174	return resource_dict
	175
158	176	def prepare(self, reactor, clock, hs):
159	177	self.user_pass = "pass"
160	178	self.user = self.register_user("test", self.user_pass)
161		self.user_tok = self.login("test", self.user_pass)
162
163		def get_device_ids(self) -> List[str]:
164		# Get the list of devices so one can be deleted.
165		request, channel = self.make_request(
166		"GET", "devices", access_token=self.user_tok,
167		) # type: SynapseRequest, FakeChannel
168
169		# Get the ID of the device.
170		self.assertEqual(request.code, 200)
171		return [d["device_id"] for d in channel.json_body["devices"]]
	179	self.device_id = "dev1"
	180	self.user_tok = self.login("test", self.user_pass, self.device_id)
172	181
173	182	def delete_device(
174		self, device: str, expected_response: int, body: Union[bytes, JsonDict] = b""
	183	self,
	184	access_token: str,
	185	device: str,
	186	expected_response: int,
	187	body: Union[bytes, JsonDict] = b"",
175	188	) -> FakeChannel:
176	189	"""Delete an individual device."""
177		request, channel = self.make_request(
178		"DELETE", "devices/" + device, body, access_token=self.user_tok
179		) # type: SynapseRequest, FakeChannel
	190	channel = self.make_request(
	191	"DELETE", "devices/" + device, body, access_token=access_token,
	192	)
180	193
181	194	# Ensure the response is sane.
182		self.assertEqual(request.code, expected_response)
	195	self.assertEqual(channel.code, expected_response)
183	196
184	197	return channel
185	198

187	200	"""Delete 1 or more devices."""
188	201	# Note that this uses the delete_devices endpoint so that we can modify
189	202	# the payload half-way through some tests.
190		request, channel = self.make_request(
	203	channel = self.make_request(
191	204	"POST", "delete_devices", body, access_token=self.user_tok,
192		) # type: SynapseRequest, FakeChannel
	205	)
193	206
194	207	# Ensure the response is sane.
195		self.assertEqual(request.code, expected_response)
	208	self.assertEqual(channel.code, expected_response)
196	209
197	210	return channel
198	211

200	213	"""
201	214	Test user interactive authentication outside of registration.
202	215	"""
203		device_id = self.get_device_ids()[0]
204
205	216	# Attempt to delete this device.
206	217	# Returns a 401 as per the spec
207		channel = self.delete_device(device_id, 401)
	218	channel = self.delete_device(self.user_tok, self.device_id, 401)
208	219
209	220	# Grab the session
210	221	session = channel.json_body["session"]

213	224
214	225	# Make another request providing the UI auth flow.
215	226	self.delete_device(
216		device_id,
	227	self.user_tok,
	228	self.device_id,
217	229	200,
218	230	{
219	231	"auth": {

232	244	UIA - check that still works.
233	245	"""
234	246
235		device_id = self.get_device_ids()[0]
236		channel = self.delete_device(device_id, 401)
	247	channel = self.delete_device(self.user_tok, self.device_id, 401)
237	248	session = channel.json_body["session"]
238	249
239	250	# Make another request providing the UI auth flow.
240	251	self.delete_device(
241		device_id,
	252	self.user_tok,
	253	self.device_id,
242	254	200,
243	255	{
244	256	"auth": {

261	273	session ID should be rejected.
262	274	"""
263	275	# Create a second login.
264		self.login("test", self.user_pass)
265
266		device_ids = self.get_device_ids()
267		self.assertEqual(len(device_ids), 2)
	276	self.login("test", self.user_pass, "dev2")
268	277
269	278	# Attempt to delete the first device.
270	279	# Returns a 401 as per the spec
271		channel = self.delete_devices(401, {"devices": [device_ids[0]]})
	280	channel = self.delete_devices(401, {"devices": [self.device_id]})
272	281
273	282	# Grab the session
274	283	session = channel.json_body["session"]

280	289	self.delete_devices(
281	290	200,
282	291	{
283		"devices": [device_ids[1]],
	292	"devices": ["dev2"],
284	293	"auth": {
285	294	"type": "m.login.password",
286	295	"identifier": {"type": "m.id.user", "user": self.user},

295	304	The initial requested URI cannot be modified during the user interactive authentication session.
296	305	"""
297	306	# Create a second login.
298		self.login("test", self.user_pass)
299
300		device_ids = self.get_device_ids()
301		self.assertEqual(len(device_ids), 2)
	307	self.login("test", self.user_pass, "dev2")
302	308
303	309	# Attempt to delete the first device.
304	310	# Returns a 401 as per the spec
305		channel = self.delete_device(device_ids[0], 401)
	311	channel = self.delete_device(self.user_tok, self.device_id, 401)
306	312
307	313	# Grab the session
308	314	session = channel.json_body["session"]

311	317
312	318	# Make another request providing the UI auth flow, but try to delete the
313	319	# second device. This results in an error.
	320	#
	321	# This makes use of the fact that the device ID is embedded into the URL.
314	322	self.delete_device(
315		device_ids[1],
	323	self.user_tok,
	324	"dev2",
316	325	403,
317	326	{
318	327	"auth": {

323	332	},
324	333	},
325	334	)
	335
	336	@unittest.override_config({"ui_auth": {"session_timeout": 5 * 1000}})
	337	def test_can_reuse_session(self):
	338	"""
	339	The session can be reused if configured.
	340
	341	Compare to test_cannot_change_uri.
	342	"""
	343	# Create a second and third login.
	344	self.login("test", self.user_pass, "dev2")
	345	self.login("test", self.user_pass, "dev3")
	346
	347	# Attempt to delete a device. This works since the user just logged in.
	348	self.delete_device(self.user_tok, "dev2", 200)
	349
	350	# Move the clock forward past the validation timeout.
	351	self.reactor.advance(6)
	352
	353	# Deleting another devices throws the user into UI auth.
	354	channel = self.delete_device(self.user_tok, "dev3", 401)
	355
	356	# Grab the session
	357	session = channel.json_body["session"]
	358	# Ensure that flows are what is expected.
	359	self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"])
	360
	361	# Make another request providing the UI auth flow.
	362	self.delete_device(
	363	self.user_tok,
	364	"dev3",
	365	200,
	366	{
	367	"auth": {
	368	"type": "m.login.password",
	369	"identifier": {"type": "m.id.user", "user": self.user},
	370	"password": self.user_pass,
	371	"session": session,
	372	},
	373	},
	374	)
	375
	376	# Make another request, but try to delete the first device. This works
	377	# due to re-using the previous session.
	378	#
	379	# Note that no auth information is provided, not even a session iD!
	380	self.delete_device(self.user_tok, self.device_id, 200)
	381
	382	def test_does_not_offer_password_for_sso_user(self):
	383	login_resp = self.helper.login_via_oidc("username")
	384	user_tok = login_resp["access_token"]
	385	device_id = login_resp["device_id"]
	386
	387	# now call the device deletion API: we should get the option to auth with SSO
	388	# and not password.
	389	channel = self.delete_device(user_tok, device_id, 401)
	390
	391	flows = channel.json_body["flows"]
	392	self.assertEqual(flows, [{"stages": ["m.login.sso"]}])
	393
	394	def test_does_not_offer_sso_for_password_user(self):
	395	# now call the device deletion API: we should get the option to auth with SSO
	396	# and not password.
	397	channel = self.delete_device(self.user_tok, self.device_id, 401)
	398
	399	flows = channel.json_body["flows"]
	400	self.assertEqual(flows, [{"stages": ["m.login.password"]}])
	401
	402	def test_offers_both_flows_for_upgraded_user(self):
	403	"""A user that had a password and then logged in with SSO should get both flows
	404	"""
	405	login_resp = self.helper.login_via_oidc(UserID.from_string(self.user).localpart)
	406	self.assertEqual(login_resp["user_id"], self.user)
	407
	408	channel = self.delete_device(self.user_tok, self.device_id, 401)
	409
	410	flows = channel.json_body["flows"]
	411	# we have no particular expectations of ordering here
	412	self.assertIn({"stages": ["m.login.password"]}, flows)
	413	self.assertIn({"stages": ["m.login.sso"]}, flows)
	414	self.assertEqual(len(flows), 2)

+4

-4

tests/rest/client/v2_alpha/test_capabilities.py less more

35	35	return hs
36	36
37	37	def test_check_auth_required(self):
38		request, channel = self.make_request("GET", self.url)
	38	channel = self.make_request("GET", self.url)
39	39
40	40	self.assertEqual(channel.code, 401)
41	41

43	43	self.register_user("user", "pass")
44	44	access_token = self.login("user", "pass")
45	45
46		request, channel = self.make_request("GET", self.url, access_token=access_token)
	46	channel = self.make_request("GET", self.url, access_token=access_token)
47	47	capabilities = channel.json_body["capabilities"]
48	48
49	49	self.assertEqual(channel.code, 200)

61	61	user = self.register_user(localpart, password)
62	62	access_token = self.login(user, password)
63	63
64		request, channel = self.make_request("GET", self.url, access_token=access_token)
	64	channel = self.make_request("GET", self.url, access_token=access_token)
65	65	capabilities = channel.json_body["capabilities"]
66	66
67	67	self.assertEqual(channel.code, 200)

69	69	# Test case where password is handled outside of Synapse
70	70	self.assertTrue(capabilities["m.change_password"]["enabled"])
71	71	self.get_success(self.store.user_set_password_hash(user, None))
72		request, channel = self.make_request("GET", self.url, access_token=access_token)
	72	channel = self.make_request("GET", self.url, access_token=access_token)
73	73	capabilities = channel.json_body["capabilities"]
74	74
75	75	self.assertEqual(channel.code, 200)

+7

-7

tests/rest/client/v2_alpha/test_filter.py less more

35	35	self.store = hs.get_datastore()
36	36
37	37	def test_add_filter(self):
38		request, channel = self.make_request(
	38	channel = self.make_request(
39	39	"POST",
40	40	"/_matrix/client/r0/user/%s/filter" % (self.user_id),
41	41	self.EXAMPLE_FILTER_JSON,

48	48	self.assertEquals(filter.result, self.EXAMPLE_FILTER)
49	49
50	50	def test_add_filter_for_other_user(self):
51		request, channel = self.make_request(
	51	channel = self.make_request(
52	52	"POST",
53	53	"/_matrix/client/r0/user/%s/filter" % ("@watermelon:test"),
54	54	self.EXAMPLE_FILTER_JSON,

60	60	def test_add_filter_non_local_user(self):
61	61	_is_mine = self.hs.is_mine
62	62	self.hs.is_mine = lambda target_user: False
63		request, channel = self.make_request(
	63	channel = self.make_request(
64	64	"POST",
65	65	"/_matrix/client/r0/user/%s/filter" % (self.user_id),
66	66	self.EXAMPLE_FILTER_JSON,

78	78	)
79	79	self.reactor.advance(1)
80	80	filter_id = filter_id.result
81		request, channel = self.make_request(
	81	channel = self.make_request(
82	82	"GET", "/_matrix/client/r0/user/%s/filter/%s" % (self.user_id, filter_id)
83	83	)
84	84

86	86	self.assertEquals(channel.json_body, self.EXAMPLE_FILTER)
87	87
88	88	def test_get_filter_non_existant(self):
89		request, channel = self.make_request(
	89	channel = self.make_request(
90	90	"GET", "/_matrix/client/r0/user/%s/filter/12382148321" % (self.user_id)
91	91	)
92	92

96	96	# Currently invalid params do not have an appropriate errcode
97	97	# in errors.py
98	98	def test_get_filter_invalid_id(self):
99		request, channel = self.make_request(
	99	channel = self.make_request(
100	100	"GET", "/_matrix/client/r0/user/%s/filter/foobar" % (self.user_id)
101	101	)
102	102

104	104
105	105	# No ID also returns an invalid_id error
106	106	def test_get_filter_no_id(self):
107		request, channel = self.make_request(
	107	channel = self.make_request(
108	108	"GET", "/_matrix/client/r0/user/%s/filter/" % (self.user_id)
109	109	)
110	110

+8

-10

tests/rest/client/v2_alpha/test_password_policy.py less more

69	69	def test_get_policy(self):
70	70	"""Tests if the /password_policy endpoint returns the configured policy."""
71	71
72		request, channel = self.make_request(
73		"GET", "/_matrix/client/r0/password_policy"
74		)
	72	channel = self.make_request("GET", "/_matrix/client/r0/password_policy")
75	73
76	74	self.assertEqual(channel.code, 200, channel.result)
77	75	self.assertEqual(

88	86
89	87	def test_password_too_short(self):
90	88	request_data = json.dumps({"username": "kermit", "password": "shorty"})
91		request, channel = self.make_request("POST", self.register_url, request_data)
	89	channel = self.make_request("POST", self.register_url, request_data)
92	90
93	91	self.assertEqual(channel.code, 400, channel.result)
94	92	self.assertEqual(

97	95
98	96	def test_password_no_digit(self):
99	97	request_data = json.dumps({"username": "kermit", "password": "longerpassword"})
100		request, channel = self.make_request("POST", self.register_url, request_data)
	98	channel = self.make_request("POST", self.register_url, request_data)
101	99
102	100	self.assertEqual(channel.code, 400, channel.result)
103	101	self.assertEqual(

106	104
107	105	def test_password_no_symbol(self):
108	106	request_data = json.dumps({"username": "kermit", "password": "l0ngerpassword"})
109		request, channel = self.make_request("POST", self.register_url, request_data)
	107	channel = self.make_request("POST", self.register_url, request_data)
110	108
111	109	self.assertEqual(channel.code, 400, channel.result)
112	110	self.assertEqual(

115	113
116	114	def test_password_no_uppercase(self):
117	115	request_data = json.dumps({"username": "kermit", "password": "l0ngerpassword!"})
118		request, channel = self.make_request("POST", self.register_url, request_data)
	116	channel = self.make_request("POST", self.register_url, request_data)
119	117
120	118	self.assertEqual(channel.code, 400, channel.result)
121	119	self.assertEqual(

124	122
125	123	def test_password_no_lowercase(self):
126	124	request_data = json.dumps({"username": "kermit", "password": "L0NGERPASSWORD!"})
127		request, channel = self.make_request("POST", self.register_url, request_data)
	125	channel = self.make_request("POST", self.register_url, request_data)
128	126
129	127	self.assertEqual(channel.code, 400, channel.result)
130	128	self.assertEqual(

133	131
134	132	def test_password_compliant(self):
135	133	request_data = json.dumps({"username": "kermit", "password": "L0ngerpassword!"})
136		request, channel = self.make_request("POST", self.register_url, request_data)
	134	channel = self.make_request("POST", self.register_url, request_data)
137	135
138	136	# Getting a 401 here means the password has passed validation and the server has
139	137	# responded with a list of registration flows.

159	157	},
160	158	}
161	159	)
162		request, channel = self.make_request(
	160	channel = self.make_request(
163	161	"POST",
164	162	"/_matrix/client/r0/account/password",
165	163	request_data,

+33

-38

tests/rest/client/v2_alpha/test_register.py less more

60	60	self.hs.get_datastore().services_cache.append(appservice)
61	61	request_data = json.dumps({"username": "as_user_kermit"})
62	62
63		request, channel = self.make_request(
	63	channel = self.make_request(
64	64	b"POST", self.url + b"?access_token=i_am_an_app_service", request_data
65	65	)
66	66

71	71	def test_POST_appservice_registration_invalid(self):
72	72	self.appservice = None # no application service exists
73	73	request_data = json.dumps({"username": "kermit"})
74		request, channel = self.make_request(
	74	channel = self.make_request(
75	75	b"POST", self.url + b"?access_token=i_am_an_app_service", request_data
76	76	)
77	77

79	79
80	80	def test_POST_bad_password(self):
81	81	request_data = json.dumps({"username": "kermit", "password": 666})
82		request, channel = self.make_request(b"POST", self.url, request_data)
	82	channel = self.make_request(b"POST", self.url, request_data)
83	83
84	84	self.assertEquals(channel.result["code"], b"400", channel.result)
85	85	self.assertEquals(channel.json_body["error"], "Invalid password")
86	86
87	87	def test_POST_bad_username(self):
88	88	request_data = json.dumps({"username": 777, "password": "monkey"})
89		request, channel = self.make_request(b"POST", self.url, request_data)
	89	channel = self.make_request(b"POST", self.url, request_data)
90	90
91	91	self.assertEquals(channel.result["code"], b"400", channel.result)
92	92	self.assertEquals(channel.json_body["error"], "Invalid username")

101	101	"auth": {"type": LoginType.DUMMY},
102	102	}
103	103	request_data = json.dumps(params)
104		request, channel = self.make_request(b"POST", self.url, request_data)
	104	channel = self.make_request(b"POST", self.url, request_data)
105	105
106	106	det_data = {
107	107	"user_id": user_id,

116	116	request_data = json.dumps({"username": "kermit", "password": "monkey"})
117	117	self.auth_result = (None, {"username": "kermit", "password": "monkey"}, None)
118	118
119		request, channel = self.make_request(b"POST", self.url, request_data)
	119	channel = self.make_request(b"POST", self.url, request_data)
120	120
121	121	self.assertEquals(channel.result["code"], b"403", channel.result)
122	122	self.assertEquals(channel.json_body["error"], "Registration has been disabled")
	123	self.assertEquals(channel.json_body["errcode"], "M_FORBIDDEN")
123	124
124	125	def test_POST_guest_registration(self):
125	126	self.hs.config.macaroon_secret_key = "test"
126	127	self.hs.config.allow_guest_access = True
127	128
128		request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	129	channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
129	130
130	131	det_data = {"home_server": self.hs.hostname, "device_id": "guest_device"}
131	132	self.assertEquals(channel.result["code"], b"200", channel.result)

134	135	def test_POST_disabled_guest_registration(self):
135	136	self.hs.config.allow_guest_access = False
136	137
137		request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	138	channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
138	139
139	140	self.assertEquals(channel.result["code"], b"403", channel.result)
140	141	self.assertEquals(channel.json_body["error"], "Guest access is disabled")

143	144	def test_POST_ratelimiting_guest(self):
144	145	for i in range(0, 6):
145	146	url = self.url + b"?kind=guest"
146		request, channel = self.make_request(b"POST", url, b"{}")
	147	channel = self.make_request(b"POST", url, b"{}")
147	148
148	149	if i == 5:
149	150	self.assertEquals(channel.result["code"], b"429", channel.result)

153	154
154	155	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
155	156
156		request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	157	channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
157	158
158	159	self.assertEquals(channel.result["code"], b"200", channel.result)
159	160

167	168	"auth": {"type": LoginType.DUMMY},
168	169	}
169	170	request_data = json.dumps(params)
170		request, channel = self.make_request(b"POST", self.url, request_data)
	171	channel = self.make_request(b"POST", self.url, request_data)
171	172
172	173	if i == 5:
173	174	self.assertEquals(channel.result["code"], b"429", channel.result)

177	178
178	179	self.reactor.advance(retry_after_ms / 1000.0 + 1.0)
179	180
180		request, channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
	181	channel = self.make_request(b"POST", self.url + b"?kind=guest", b"{}")
181	182
182	183	self.assertEquals(channel.result["code"], b"200", channel.result)
183	184
184	185	def test_advertised_flows(self):
185		request, channel = self.make_request(b"POST", self.url, b"{}")
	186	channel = self.make_request(b"POST", self.url, b"{}")
186	187	self.assertEquals(channel.result["code"], b"401", channel.result)
187	188	flows = channel.json_body["flows"]
188	189

205	206	}
206	207	)
207	208	def test_advertised_flows_captcha_and_terms_and_3pids(self):
208		request, channel = self.make_request(b"POST", self.url, b"{}")
	209	channel = self.make_request(b"POST", self.url, b"{}")
209	210	self.assertEquals(channel.result["code"], b"401", channel.result)
210	211	flows = channel.json_body["flows"]
211	212

237	238	}
238	239	)
239	240	def test_advertised_flows_no_msisdn_email_required(self):
240		request, channel = self.make_request(b"POST", self.url, b"{}")
	241	channel = self.make_request(b"POST", self.url, b"{}")
241	242	self.assertEquals(channel.result["code"], b"401", channel.result)
242	243	flows = channel.json_body["flows"]
243	244

277	278	)
278	279	)
279	280
280		request, channel = self.make_request(
	281	channel = self.make_request(
281	282	"POST",
282	283	b"register/email/requestToken",
283	284	{"client_secret": "foobar", "email": email, "send_attempt": 1},

316	317
317	318	# The specific endpoint doesn't matter, all we need is an authenticated
318	319	# endpoint.
319		request, channel = self.make_request(b"GET", "/sync", access_token=tok)
	320	channel = self.make_request(b"GET", "/sync", access_token=tok)
320	321
321	322	self.assertEquals(channel.result["code"], b"200", channel.result)
322	323
323	324	self.reactor.advance(datetime.timedelta(weeks=1).total_seconds())
324	325
325		request, channel = self.make_request(b"GET", "/sync", access_token=tok)
	326	channel = self.make_request(b"GET", "/sync", access_token=tok)
326	327
327	328	self.assertEquals(channel.result["code"], b"403", channel.result)
328	329	self.assertEquals(

344	345	url = "/_synapse/admin/v1/account_validity/validity"
345	346	params = {"user_id": user_id}
346	347	request_data = json.dumps(params)
347		request, channel = self.make_request(
348		b"POST", url, request_data, access_token=admin_tok
349		)
	348	channel = self.make_request(b"POST", url, request_data, access_token=admin_tok)
350	349	self.assertEquals(channel.result["code"], b"200", channel.result)
351	350
352	351	# The specific endpoint doesn't matter, all we need is an authenticated
353	352	# endpoint.
354		request, channel = self.make_request(b"GET", "/sync", access_token=tok)
	353	channel = self.make_request(b"GET", "/sync", access_token=tok)
355	354	self.assertEquals(channel.result["code"], b"200", channel.result)
356	355
357	356	def test_manual_expire(self):

368	367	"enable_renewal_emails": False,
369	368	}
370	369	request_data = json.dumps(params)
371		request, channel = self.make_request(
372		b"POST", url, request_data, access_token=admin_tok
373		)
	370	channel = self.make_request(b"POST", url, request_data, access_token=admin_tok)
374	371	self.assertEquals(channel.result["code"], b"200", channel.result)
375	372
376	373	# The specific endpoint doesn't matter, all we need is an authenticated
377	374	# endpoint.
378		request, channel = self.make_request(b"GET", "/sync", access_token=tok)
	375	channel = self.make_request(b"GET", "/sync", access_token=tok)
379	376	self.assertEquals(channel.result["code"], b"403", channel.result)
380	377	self.assertEquals(
381	378	channel.json_body["errcode"], Codes.EXPIRED_ACCOUNT, channel.result

395	392	"enable_renewal_emails": False,
396	393	}
397	394	request_data = json.dumps(params)
398		request, channel = self.make_request(
399		b"POST", url, request_data, access_token=admin_tok
400		)
	395	channel = self.make_request(b"POST", url, request_data, access_token=admin_tok)
401	396	self.assertEquals(channel.result["code"], b"200", channel.result)
402	397
403	398	# Try to log the user out
404		request, channel = self.make_request(b"POST", "/logout", access_token=tok)
	399	channel = self.make_request(b"POST", "/logout", access_token=tok)
405	400	self.assertEquals(channel.result["code"], b"200", channel.result)
406	401
407	402	# Log the user in again (allowed for expired accounts)
408	403	tok = self.login("kermit", "monkey")
409	404
410	405	# Try to log out all of the user's sessions
411		request, channel = self.make_request(b"POST", "/logout/all", access_token=tok)
	406	channel = self.make_request(b"POST", "/logout/all", access_token=tok)
412	407	self.assertEquals(channel.result["code"], b"200", channel.result)
413	408
414	409

482	477	# retrieve the token from the DB.
483	478	renewal_token = self.get_success(self.store.get_renewal_token_for_user(user_id))
484	479	url = "/_matrix/client/unstable/account_validity/renew?token=%s" % renewal_token
485		request, channel = self.make_request(b"GET", url)
	480	channel = self.make_request(b"GET", url)
486	481	self.assertEquals(channel.result["code"], b"200", channel.result)
487	482
488	483	# Check that we're getting HTML back.

502	497	# our access token should be denied from now, otherwise they should
503	498	# succeed.
504	499	self.reactor.advance(datetime.timedelta(days=3).total_seconds())
505		request, channel = self.make_request(b"GET", "/sync", access_token=tok)
	500	channel = self.make_request(b"GET", "/sync", access_token=tok)
506	501	self.assertEquals(channel.result["code"], b"200", channel.result)
507	502
508	503	def test_renewal_invalid_token(self):
509	504	# Hit the renewal endpoint with an invalid token and check that it behaves as
510	505	# expected, i.e. that it responds with 404 Not Found and the correct HTML.
511	506	url = "/_matrix/client/unstable/account_validity/renew?token=123"
512		request, channel = self.make_request(b"GET", url)
	507	channel = self.make_request(b"GET", url)
513	508	self.assertEquals(channel.result["code"], b"404", channel.result)
514	509
515	510	# Check that we're getting HTML back.

530	525	self.email_attempts = []
531	526
532	527	(user_id, tok) = self.create_user()
533		request, channel = self.make_request(
	528	channel = self.make_request(
534	529	b"POST",
535	530	"/_matrix/client/unstable/account_validity/send_mail",
536	531	access_token=tok,

554	549	"erase": False,
555	550	}
556	551	)
557		request, channel = self.make_request(
	552	channel = self.make_request(
558	553	"POST", "account/deactivate", request_data, access_token=tok
559	554	)
560		self.assertEqual(request.code, 200)
	555	self.assertEqual(channel.code, 200)
561	556
562	557	self.reactor.advance(datetime.timedelta(days=8).total_seconds())
563	558

605	600	self.email_attempts = []
606	601
607	602	# Test that we're still able to manually trigger a mail to be sent.
608		request, channel = self.make_request(
	603	channel = self.make_request(
609	604	b"POST",
610	605	"/_matrix/client/unstable/account_validity/send_mail",
611	606	access_token=tok,

+18

-18

tests/rest/client/v2_alpha/test_relations.py less more

59	59
60	60	event_id = channel.json_body["event_id"]
61	61
62		request, channel = self.make_request(
	62	channel = self.make_request(
63	63	"GET",
64	64	"/rooms/%s/event/%s" % (self.room, event_id),
65	65	access_token=self.user_token,

106	106	self.assertEquals(200, channel.code, channel.json_body)
107	107	annotation_id = channel.json_body["event_id"]
108	108
109		request, channel = self.make_request(
	109	channel = self.make_request(
110	110	"GET",
111	111	"/_matrix/client/unstable/rooms/%s/relations/%s?limit=1"
112	112	% (self.room, self.parent_id),

151	151	if prev_token:
152	152	from_token = "&from=" + prev_token
153	153
154		request, channel = self.make_request(
	154	channel = self.make_request(
155	155	"GET",
156	156	"/_matrix/client/unstable/rooms/%s/relations/%s?limit=1%s"
157	157	% (self.room, self.parent_id, from_token),

209	209	if prev_token:
210	210	from_token = "&from=" + prev_token
211	211
212		request, channel = self.make_request(
	212	channel = self.make_request(
213	213	"GET",
214	214	"/_matrix/client/unstable/rooms/%s/aggregations/%s?limit=1%s"
215	215	% (self.room, self.parent_id, from_token),

278	278	if prev_token:
279	279	from_token = "&from=" + prev_token
280	280
281		request, channel = self.make_request(
	281	channel = self.make_request(
282	282	"GET",
283	283	"/_matrix/client/unstable/rooms/%s"
284	284	"/aggregations/%s/%s/m.reaction/%s?limit=1%s"

324	324	channel = self._send_relation(RelationTypes.ANNOTATION, "m.reaction", "b")
325	325	self.assertEquals(200, channel.code, channel.json_body)
326	326
327		request, channel = self.make_request(
	327	channel = self.make_request(
328	328	"GET",
329	329	"/_matrix/client/unstable/rooms/%s/aggregations/%s"
330	330	% (self.room, self.parent_id),

356	356	self.assertEquals(200, channel.code, channel.json_body)
357	357
358	358	# Now lets redact one of the 'a' reactions
359		request, channel = self.make_request(
	359	channel = self.make_request(
360	360	"POST",
361	361	"/_matrix/client/r0/rooms/%s/redact/%s" % (self.room, to_redact_event_id),
362	362	access_token=self.user_token,

364	364	)
365	365	self.assertEquals(200, channel.code, channel.json_body)
366	366
367		request, channel = self.make_request(
	367	channel = self.make_request(
368	368	"GET",
369	369	"/_matrix/client/unstable/rooms/%s/aggregations/%s"
370	370	% (self.room, self.parent_id),

381	381	"""Test that aggregations must be annotations.
382	382	"""
383	383
384		request, channel = self.make_request(
	384	channel = self.make_request(
385	385	"GET",
386	386	"/_matrix/client/unstable/rooms/%s/aggregations/%s/%s?limit=1"
387	387	% (self.room, self.parent_id, RelationTypes.REPLACE),

413	413	self.assertEquals(200, channel.code, channel.json_body)
414	414	reply_2 = channel.json_body["event_id"]
415	415
416		request, channel = self.make_request(
	416	channel = self.make_request(
417	417	"GET",
418	418	"/rooms/%s/event/%s" % (self.room, self.parent_id),
419	419	access_token=self.user_token,

449	449
450	450	edit_event_id = channel.json_body["event_id"]
451	451
452		request, channel = self.make_request(
	452	channel = self.make_request(
453	453	"GET",
454	454	"/rooms/%s/event/%s" % (self.room, self.parent_id),
455	455	access_token=self.user_token,

506	506	)
507	507	self.assertEquals(200, channel.code, channel.json_body)
508	508
509		request, channel = self.make_request(
	509	channel = self.make_request(
510	510	"GET",
511	511	"/rooms/%s/event/%s" % (self.room, self.parent_id),
512	512	access_token=self.user_token,

548	548	self.assertEquals(200, channel.code, channel.json_body)
549	549
550	550	# Check the relation is returned
551		request, channel = self.make_request(
	551	channel = self.make_request(
552	552	"GET",
553	553	"/_matrix/client/unstable/rooms/%s/relations/%s/m.replace/m.room.message"
554	554	% (self.room, original_event_id),

560	560	self.assertEquals(len(channel.json_body["chunk"]), 1)
561	561
562	562	# Redact the original event
563		request, channel = self.make_request(
	563	channel = self.make_request(
564	564	"PUT",
565	565	"/rooms/%s/redact/%s/%s"
566	566	% (self.room, original_event_id, "test_relations_redaction_redacts_edits"),

570	570	self.assertEquals(200, channel.code, channel.json_body)
571	571
572	572	# Try to check for remaining m.replace relations
573		request, channel = self.make_request(
	573	channel = self.make_request(
574	574	"GET",
575	575	"/_matrix/client/unstable/rooms/%s/relations/%s/m.replace/m.room.message"
576	576	% (self.room, original_event_id),

597	597	self.assertEquals(200, channel.code, channel.json_body)
598	598
599	599	# Redact the original
600		request, channel = self.make_request(
	600	channel = self.make_request(
601	601	"PUT",
602	602	"/rooms/%s/redact/%s/%s"
603	603	% (

611	611	self.assertEquals(200, channel.code, channel.json_body)
612	612
613	613	# Check that aggregations returns zero
614		request, channel = self.make_request(
	614	channel = self.make_request(
615	615	"GET",
616	616	"/_matrix/client/unstable/rooms/%s/aggregations/%s/m.annotation/m.reaction"
617	617	% (self.room, original_event_id),

655	655
656	656	original_id = parent_id if parent_id else self.parent_id
657	657
658		request, channel = self.make_request(
	658	channel = self.make_request(
659	659	"POST",
660	660	"/_matrix/client/unstable/rooms/%s/send_relation/%s/%s/%s%s"
661	661	% (self.room, original_id, relation_type, event_type, query),

+8

-8

tests/rest/client/v2_alpha/test_shared_rooms.py less more

16	16	from synapse.rest.client.v2_alpha import shared_rooms
17	17
18	18	from tests import unittest
	19	from tests.server import FakeChannel
19	20
20	21
21	22	class UserSharedRoomsTest(unittest.HomeserverTestCase):

39	40	self.store = hs.get_datastore()
40	41	self.handler = hs.get_user_directory_handler()
41	42
42		def _get_shared_rooms(self, token, other_user):
43		request, channel = self.make_request(
	43	def _get_shared_rooms(self, token, other_user) -> FakeChannel:
	44	return self.make_request(
44	45	"GET",
45	46	"/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/%s"
46	47	% other_user,
47	48	access_token=token,
48	49	)
49		return request, channel
50	50
51	51	def test_shared_room_list_public(self):
52	52	"""

62	62	self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
63	63	self.helper.join(room, user=u2, tok=u2_token)
64	64
65		request, channel = self._get_shared_rooms(u1_token, u2)
	65	channel = self._get_shared_rooms(u1_token, u2)
66	66	self.assertEquals(200, channel.code, channel.result)
67	67	self.assertEquals(len(channel.json_body["joined"]), 1)
68	68	self.assertEquals(channel.json_body["joined"][0], room)

81	81	self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
82	82	self.helper.join(room, user=u2, tok=u2_token)
83	83
84		request, channel = self._get_shared_rooms(u1_token, u2)
	84	channel = self._get_shared_rooms(u1_token, u2)
85	85	self.assertEquals(200, channel.code, channel.result)
86	86	self.assertEquals(len(channel.json_body["joined"]), 1)
87	87	self.assertEquals(channel.json_body["joined"][0], room)

103	103	self.helper.join(room_public, user=u2, tok=u2_token)
104	104	self.helper.join(room_private, user=u1, tok=u1_token)
105	105
106		request, channel = self._get_shared_rooms(u1_token, u2)
	106	channel = self._get_shared_rooms(u1_token, u2)
107	107	self.assertEquals(200, channel.code, channel.result)
108	108	self.assertEquals(len(channel.json_body["joined"]), 2)
109	109	self.assertTrue(room_public in channel.json_body["joined"])

124	124	self.helper.join(room, user=u2, tok=u2_token)
125	125
126	126	# Assert user directory is not empty
127		request, channel = self._get_shared_rooms(u1_token, u2)
	127	channel = self._get_shared_rooms(u1_token, u2)
128	128	self.assertEquals(200, channel.code, channel.result)
129	129	self.assertEquals(len(channel.json_body["joined"]), 1)
130	130	self.assertEquals(channel.json_body["joined"][0], room)
131	131
132	132	self.helper.leave(room, user=u1, tok=u1_token)
133	133
134		request, channel = self._get_shared_rooms(u2_token, u1)
	134	channel = self._get_shared_rooms(u2_token, u1)
135	135	self.assertEquals(200, channel.code, channel.result)
136	136	self.assertEquals(len(channel.json_body["joined"]), 0)

+12

-20

tests/rest/client/v2_alpha/test_sync.py less more

34	34	]
35	35
36	36	def test_sync_argless(self):
37		request, channel = self.make_request("GET", "/sync")
	37	channel = self.make_request("GET", "/sync")
38	38
39	39	self.assertEqual(channel.code, 200)
40	40	self.assertTrue(

54	54	"""
55	55	self.hs.config.use_presence = False
56	56
57		request, channel = self.make_request("GET", "/sync")
	57	channel = self.make_request("GET", "/sync")
58	58
59	59	self.assertEqual(channel.code, 200)
60	60	self.assertTrue(

193	193	tok=tok,
194	194	)
195	195
196		request, channel = self.make_request(
	196	channel = self.make_request(
197	197	"GET", "/sync?filter=%s" % sync_filter, access_token=tok
198	198	)
199	199	self.assertEqual(channel.code, 200, channel.result)

244	244	self.helper.send(room, body="There!", tok=other_access_token)
245	245
246	246	# Start typing.
247		request, channel = self.make_request(
	247	channel = self.make_request(
248	248	"PUT",
249	249	typing_url % (room, other_user_id, other_access_token),
250	250	b'{"typing": true, "timeout": 30000}',
251	251	)
252	252	self.assertEquals(200, channel.code)
253	253
254		request, channel = self.make_request(
255		"GET", "/sync?access_token=%s" % (access_token,)
256		)
	254	channel = self.make_request("GET", "/sync?access_token=%s" % (access_token,))
257	255	self.assertEquals(200, channel.code)
258	256	next_batch = channel.json_body["next_batch"]
259	257
260	258	# Stop typing.
261		request, channel = self.make_request(
	259	channel = self.make_request(
262	260	"PUT",
263	261	typing_url % (room, other_user_id, other_access_token),
264	262	b'{"typing": false}',

266	264	self.assertEquals(200, channel.code)
267	265
268	266	# Start typing.
269		request, channel = self.make_request(
	267	channel = self.make_request(
270	268	"PUT",
271	269	typing_url % (room, other_user_id, other_access_token),
272	270	b'{"typing": true, "timeout": 30000}',

274	272	self.assertEquals(200, channel.code)
275	273
276	274	# Should return immediately
277		request, channel = self.make_request(
278		"GET", sync_url % (access_token, next_batch)
279		)
	275	channel = self.make_request("GET", sync_url % (access_token, next_batch))
280	276	self.assertEquals(200, channel.code)
281	277	next_batch = channel.json_body["next_batch"]
282	278

288	284	# invalidate the stream token.
289	285	self.helper.send(room, body="There!", tok=other_access_token)
290	286
291		request, channel = self.make_request(
292		"GET", sync_url % (access_token, next_batch)
293		)
	287	channel = self.make_request("GET", sync_url % (access_token, next_batch))
294	288	self.assertEquals(200, channel.code)
295	289	next_batch = channel.json_body["next_batch"]
296	290

298	292	# ahead, and therefore it's saying the typing (that we've actually
299	293	# already seen) is new, since it's got a token above our new, now-reset
300	294	# stream token.
301		request, channel = self.make_request(
302		"GET", sync_url % (access_token, next_batch)
303		)
	295	channel = self.make_request("GET", sync_url % (access_token, next_batch))
304	296	self.assertEquals(200, channel.code)
305	297	next_batch = channel.json_body["next_batch"]
306	298

382	374
383	375	# Send a read receipt to tell the server we've read the latest event.
384	376	body = json.dumps({"m.read": res["event_id"]}).encode("utf8")
385		request, channel = self.make_request(
	377	channel = self.make_request(
386	378	"POST",
387	379	"/rooms/%s/read_markers" % self.room_id,
388	380	body,

449	441	def _check_unread_count(self, expected_count: True):
450	442	"""Syncs and compares the unread count with the expected value."""
451	443
452		request, channel = self.make_request(
	444	channel = self.make_request(
453	445	"GET", self.url % self.next_batch, access_token=self.tok,
454	446	)
455	447

+2

-2

tests/rest/key/v2/test_remote_key_resource.py less more

38	38	class BaseRemoteKeyResourceTestCase(unittest.HomeserverTestCase):
39	39	def make_homeserver(self, reactor, clock):
40	40	self.http_client = Mock()
41		return self.setup_test_homeserver(http_client=self.http_client)
	41	return self.setup_test_homeserver(federation_http_client=self.http_client)
42	42
43	43	def create_test_resource(self):
44	44	return create_resource_tree(

171	171	}
172	172	]
173	173	self.hs2 = self.setup_test_homeserver(
174		http_client=self.http_client2, config=config
	174	federation_http_client=self.http_client2, config=config
175	175	)
176	176
177	177	# wire up outbound POST /key/v2/query requests from hs2 so that they

+16

-3

tests/rest/media/v1/test_media_storage.py less more

213	213	}
214	214	config["media_storage_providers"] = [provider_config]
215	215
216		hs = self.setup_test_homeserver(config=config, http_client=client)
	216	hs = self.setup_test_homeserver(config=config, federation_http_client=client)
217	217
218	218	return hs
219	219

227	227
228	228	def _req(self, content_disposition):
229	229
230		request, channel = make_request(
	230	channel = make_request(
231	231	self.reactor,
232	232	FakeSite(self.download_resource),
233	233	"GET",

323	323
324	324	def _test_thumbnail(self, method, expected_body, expected_found):
325	325	params = "?width=32&height=32&method=" + method
326		request, channel = make_request(
	326	channel = make_request(
327	327	self.reactor,
328	328	FakeSite(self.thumbnail_resource),
329	329	"GET",

361	361	"error": "Not found [b'example.com', b'12345']",
362	362	},
363	363	)
	364
	365	def test_x_robots_tag_header(self):
	366	"""
	367	Tests that the `X-Robots-Tag` header is present, which informs web crawlers
	368	to not index, archive, or follow links in media.
	369	"""
	370	channel = self._req(b"inline; filename=out" + self.test_image.extension)
	371
	372	headers = channel.headers
	373	self.assertEqual(
	374	headers.getRawHeaders(b"X-Robots-Tag"),
	375	[b"noindex, nofollow, noarchive, noimageindex"],
	376	)

+19

-45

tests/rest/media/v1/test_url_preview.py less more

17	17
18	18	from mock import patch
19	19
20		import attr
21
22	20	from twisted.internet._resolver import HostResolution
23	21	from twisted.internet.address import IPv4Address, IPv6Address
24	22	from twisted.internet.error import DNSLookupError
25		from twisted.python.failure import Failure
26	23	from twisted.test.proto_helpers import AccumulatingProtocol
27		from twisted.web._newclient import ResponseDone
28	24
29	25	from tests import unittest
30	26	from tests.server import FakeTransport
31
32
33		@attr.s
34		class FakeResponse:
35		version = attr.ib()
36		code = attr.ib()
37		phrase = attr.ib()
38		headers = attr.ib()
39		body = attr.ib()
40		absoluteURI = attr.ib()
41
42		@property
43		def request(self):
44		@attr.s
45		class FakeTransport:
46		absoluteURI = self.absoluteURI
47
48		return FakeTransport()
49
50		def deliverBody(self, protocol):
51		protocol.dataReceived(self.body)
52		protocol.connectionLost(Failure(ResponseDone()))
53	27
54	28
55	29	class URLPreviewTests(unittest.HomeserverTestCase):

138	112	def test_cache_returns_correct_type(self):
139	113	self.lookups["matrix.org"] = [(IPv4Address, "10.1.2.3")]
140	114
141		request, channel = self.make_request(
	115	channel = self.make_request(
142	116	"GET",
143	117	"preview_url?url=http://matrix.org",
144	118	shorthand=False,

163	137	)
164	138
165	139	# Check the cache returns the correct response
166		request, channel = self.make_request(
	140	channel = self.make_request(
167	141	"GET", "preview_url?url=http://matrix.org", shorthand=False
168	142	)
169	143

179	153	self.assertNotIn("http://matrix.org", self.preview_url._cache)
180	154
181	155	# Check the database cache returns the correct response
182		request, channel = self.make_request(
	156	channel = self.make_request(
183	157	"GET", "preview_url?url=http://matrix.org", shorthand=False
184	158	)
185	159

200	174	b"</head></html>"
201	175	)
202	176
203		request, channel = self.make_request(
	177	channel = self.make_request(
204	178	"GET",
205	179	"preview_url?url=http://matrix.org",
206	180	shorthand=False,

235	209	b"</head></html>"
236	210	)
237	211
238		request, channel = self.make_request(
	212	channel = self.make_request(
239	213	"GET",
240	214	"preview_url?url=http://matrix.org",
241	215	shorthand=False,

270	244	b"</head></html>"
271	245	)
272	246
273		request, channel = self.make_request(
	247	channel = self.make_request(
274	248	"GET",
275	249	"preview_url?url=http://matrix.org",
276	250	shorthand=False,

303	277	"""
304	278	self.lookups["example.com"] = [(IPv4Address, "10.1.2.3")]
305	279
306		request, channel = self.make_request(
	280	channel = self.make_request(
307	281	"GET",
308	282	"preview_url?url=http://example.com",
309	283	shorthand=False,

333	307	"""
334	308	self.lookups["example.com"] = [(IPv4Address, "192.168.1.1")]
335	309
336		request, channel = self.make_request(
	310	channel = self.make_request(
337	311	"GET", "preview_url?url=http://example.com", shorthand=False
338	312	)
339	313

354	328	"""
355	329	self.lookups["example.com"] = [(IPv4Address, "1.1.1.2")]
356	330
357		request, channel = self.make_request(
	331	channel = self.make_request(
358	332	"GET", "preview_url?url=http://example.com", shorthand=False
359	333	)
360	334

371	345	"""
372	346	Blacklisted IP addresses, accessed directly, are not spidered.
373	347	"""
374		request, channel = self.make_request(
	348	channel = self.make_request(
375	349	"GET", "preview_url?url=http://192.168.1.1", shorthand=False
376	350	)
377	351

390	364	"""
391	365	Blacklisted IP ranges, accessed directly, are not spidered.
392	366	"""
393		request, channel = self.make_request(
	367	channel = self.make_request(
394	368	"GET", "preview_url?url=http://1.1.1.2", shorthand=False
395	369	)
396	370

410	384	"""
411	385	self.lookups["example.com"] = [(IPv4Address, "1.1.1.1")]
412	386
413		request, channel = self.make_request(
	387	channel = self.make_request(
414	388	"GET",
415	389	"preview_url?url=http://example.com",
416	390	shorthand=False,

447	421	(IPv4Address, "10.1.2.3"),
448	422	]
449	423
450		request, channel = self.make_request(
	424	channel = self.make_request(
451	425	"GET", "preview_url?url=http://example.com", shorthand=False
452	426	)
453	427	self.assertEqual(channel.code, 502)

467	441	(IPv6Address, "3fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff")
468	442	]
469	443
470		request, channel = self.make_request(
	444	channel = self.make_request(
471	445	"GET", "preview_url?url=http://example.com", shorthand=False
472	446	)
473	447

488	462	"""
489	463	self.lookups["example.com"] = [(IPv6Address, "2001:800::1")]
490	464
491		request, channel = self.make_request(
	465	channel = self.make_request(
492	466	"GET", "preview_url?url=http://example.com", shorthand=False
493	467	)
494	468

505	479	"""
506	480	OPTIONS returns the OPTIONS.
507	481	"""
508		request, channel = self.make_request(
	482	channel = self.make_request(
509	483	"OPTIONS", "preview_url?url=http://example.com", shorthand=False
510	484	)
511	485	self.assertEqual(channel.code, 200)

518	492	self.lookups["example.com"] = [(IPv4Address, "10.1.2.3")]
519	493
520	494	# Build and make a request to the server
521		request, channel = self.make_request(
	495	channel = self.make_request(
522	496	"GET",
523	497	"preview_url?url=http://example.com",
524	498	shorthand=False,

592	566	b"</head></html>"
593	567	)
594	568
595		request, channel = self.make_request(
	569	channel = self.make_request(
596	570	"GET",
597	571	"preview_url?url=http://twitter.com/matrixdotorg/status/12345",
598	572	shorthand=False,

657	631	}
658	632	end_content = json.dumps(result).encode("utf-8")
659	633
660		request, channel = self.make_request(
	634	channel = self.make_request(
661	635	"GET",
662	636	"preview_url?url=http://twitter.com/matrixdotorg/status/12345",
663	637	shorthand=False,

+2

-2

tests/rest/test_health.py less more

24	24	return HealthResource()
25	25
26	26	def test_health(self):
27		request, channel = self.make_request("GET", "/health", shorthand=False)
	27	channel = self.make_request("GET", "/health", shorthand=False)
28	28
29		self.assertEqual(request.code, 200)
	29	self.assertEqual(channel.code, 200)
30	30	self.assertEqual(channel.result["body"], b"OK")

+4

-4

tests/rest/test_well_known.py less more

27	27	self.hs.config.public_baseurl = "https://tesths"
28	28	self.hs.config.default_identity_server = "https://testis"
29	29
30		request, channel = self.make_request(
	30	channel = self.make_request(
31	31	"GET", "/.well-known/matrix/client", shorthand=False
32	32	)
33	33
34		self.assertEqual(request.code, 200)
	34	self.assertEqual(channel.code, 200)
35	35	self.assertEqual(
36	36	channel.json_body,
37	37	{

43	43	def test_well_known_no_public_baseurl(self):
44	44	self.hs.config.public_baseurl = None
45	45
46		request, channel = self.make_request(
	46	channel = self.make_request(
47	47	"GET", "/.well-known/matrix/client", shorthand=False
48	48	)
49	49
50		self.assertEqual(request.code, 404)
	50	self.assertEqual(channel.code, 404)

+7

-5

tests/server.py less more

173	173	custom_headers: Optional[
174	174	Iterable[Tuple[Union[bytes, str], Union[bytes, str]]]
175	175	] = None,
176		):
	176	) -> FakeChannel:
177	177	"""
178	178	Make a web request using the given method, path and content, and render it
179	179
180		Returns the Request and the Channel underneath.
	180	Returns the fake Channel object which records the response to the request.
181	181
182	182	Args:
183	183	site: The twisted Site to use to render the request

201	201	is finished.
202	202
203	203	Returns:
204		Tuple[synapse.http.site.SynapseRequest, channel]
	204	channel
205	205	"""
206	206	if not isinstance(method, bytes):
207	207	method = method.encode("ascii")

215	215	and not path.startswith(b"/_matrix")
216	216	and not path.startswith(b"/_synapse")
217	217	):
	218	if path.startswith(b"/"):
	219	path = path[1:]
218	220	path = b"/_matrix/client/r0/" + path
219		path = path.replace(b"//", b"/")
220	221
221	222	if not path.startswith(b"/"):
222	223	path = b"/" + path

257	258	for k, v in custom_headers:
258	259	req.requestHeaders.addRawHeader(k, v)
259	260
	261	req.parseCookies()
260	262	req.requestReceived(method, path, b"1.1")
261	263
262	264	if await_result:
263	265	channel.await_result()
264	266
265		return req, channel
	267	return channel
266	268
267	269
268	270	@implementer(IReactorPluggableNameResolver)

+3

-3

tests/server_notices/test_consent.py less more

69	69	the notice URL + an authentication code.
70	70	"""
71	71	# Initial sync, to get the user consent room invite
72		request, channel = self.make_request(
	72	channel = self.make_request(
73	73	"GET", "/_matrix/client/r0/sync", access_token=self.access_token
74	74	)
75	75	self.assertEqual(channel.code, 200)

78	78	room_id = list(channel.json_body["rooms"]["invite"].keys())[0]
79	79
80	80	# Join the room
81		request, channel = self.make_request(
	81	channel = self.make_request(
82	82	"POST",
83	83	"/_matrix/client/r0/rooms/" + room_id + "/join",
84	84	access_token=self.access_token,

86	86	self.assertEqual(channel.code, 200)
87	87
88	88	# Sync again, to get the message in the room
89		request, channel = self.make_request(
	89	channel = self.make_request(
90	90	"GET", "/_matrix/client/r0/sync", access_token=self.access_token
91	91	)
92	92	self.assertEqual(channel.code, 200)

+3

-5

tests/server_notices/test_resource_limits_server_notices.py less more

304	304	self.register_user("user", "password")
305	305	tok = self.login("user", "password")
306	306
307		request, channel = self.make_request("GET", "/sync?timeout=0", access_token=tok)
	307	channel = self.make_request("GET", "/sync?timeout=0", access_token=tok)
308	308
309	309	invites = channel.json_body["rooms"]["invite"]
310	310	self.assertEqual(len(invites), 0, invites)

317	317
318	318	# Sync again to retrieve the events in the room, so we can check whether this
319	319	# room has a notice in it.
320		request, channel = self.make_request("GET", "/sync?timeout=0", access_token=tok)
	320	channel = self.make_request("GET", "/sync?timeout=0", access_token=tok)
321	321
322	322	# Scan the events in the room to search for a message from the server notices
323	323	# user.

352	352	tok = self.login(localpart, "password")
353	353
354	354	# Sync with the user's token to mark the user as active.
355		request, channel = self.make_request(
356		"GET", "/sync?timeout=0", access_token=tok,
357		)
	355	channel = self.make_request("GET", "/sync?timeout=0", access_token=tok,)
358	356
359	357	# Also retrieves the list of invites for this user. We don't care about that
360	358	# one except if we're processing the last user, which should have received an

+190

-3

tests/state/test_v2.py less more

23	23	from synapse.api.room_versions import RoomVersions
24	24	from synapse.event_auth import auth_types_for_event
25	25	from synapse.events import make_event_from_dict
26		from synapse.state.v2 import lexicographical_topological_sort, resolve_events_with_store
	26	from synapse.state.v2 import (
	27	_get_auth_chain_difference,
	28	lexicographical_topological_sort,
	29	resolve_events_with_store,
	30	)
27	31	from synapse.types import EventID
28	32
29	33	from tests import unittest

83	87	event_dict = {
84	88	"auth_events": [(a, {}) for a in auth_events],
85	89	"prev_events": [(p, {}) for p in prev_events],
86		"event_id": self.node_id,
	90	"event_id": self.event_id,
87	91	"sender": self.sender,
88	92	"type": self.type,
89	93	"content": self.content,

376	380
377	381	self.do_check(events, edges, expected_state_ids)
378	382
	383	def test_mainline_sort(self):
	384	"""Tests that the mainline ordering works correctly.
	385	"""
	386
	387	events = [
	388	FakeEvent(
	389	id="T1", sender=ALICE, type=EventTypes.Topic, state_key="", content={}
	390	),
	391	FakeEvent(
	392	id="PA1",
	393	sender=ALICE,
	394	type=EventTypes.PowerLevels,
	395	state_key="",
	396	content={"users": {ALICE: 100, BOB: 50}},
	397	),
	398	FakeEvent(
	399	id="T2", sender=ALICE, type=EventTypes.Topic, state_key="", content={}
	400	),
	401	FakeEvent(
	402	id="PA2",
	403	sender=ALICE,
	404	type=EventTypes.PowerLevels,
	405	state_key="",
	406	content={
	407	"users": {ALICE: 100, BOB: 50},
	408	"events": {EventTypes.PowerLevels: 100},
	409	},
	410	),
	411	FakeEvent(
	412	id="PB",
	413	sender=BOB,
	414	type=EventTypes.PowerLevels,
	415	state_key="",
	416	content={"users": {ALICE: 100, BOB: 50}},
	417	),
	418	FakeEvent(
	419	id="T3", sender=BOB, type=EventTypes.Topic, state_key="", content={}
	420	),
	421	FakeEvent(
	422	id="T4", sender=ALICE, type=EventTypes.Topic, state_key="", content={}
	423	),
	424	]
	425
	426	edges = [
	427	["END", "T3", "PA2", "T2", "PA1", "T1", "START"],
	428	["END", "T4", "PB", "PA1"],
	429	]
	430
	431	# We expect T3 to be picked as the other topics are pointing at older
	432	# power levels. Note that without mainline ordering we'd pick T4 due to
	433	# it being sent after T3.
	434	expected_state_ids = ["T3", "PA2"]
	435
	436	self.do_check(events, edges, expected_state_ids)
	437
379	438	def do_check(self, events, edges, expected_state_ids):
380	439	"""Take a list of events and edges and calculate the state of the
381	440	graph at END, and asserts it matches `expected_state_ids`

586	645	self.assert_dict(self.expected_combined_state, state)
587	646
588	647
	648	class AuthChainDifferenceTestCase(unittest.TestCase):
	649	"""We test that `_get_auth_chain_difference` correctly handles unpersisted
	650	events.
	651	"""
	652
	653	def test_simple(self):
	654	# Test getting the auth difference for a simple chain with a single
	655	# unpersisted event:
	656	#
	657	# Unpersisted \| Persisted
	658	# \|
	659	# C -\|-> B -> A
	660
	661	a = FakeEvent(
	662	id="A", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	663	).to_event([], [])
	664
	665	b = FakeEvent(
	666	id="B", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	667	).to_event([a.event_id], [])
	668
	669	c = FakeEvent(
	670	id="C", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	671	).to_event([b.event_id], [])
	672
	673	persisted_events = {a.event_id: a, b.event_id: b}
	674	unpersited_events = {c.event_id: c}
	675
	676	state_sets = [{"a": a.event_id, "b": b.event_id}, {"c": c.event_id}]
	677
	678	store = TestStateResolutionStore(persisted_events)
	679
	680	diff_d = _get_auth_chain_difference(
	681	ROOM_ID, state_sets, unpersited_events, store
	682	)
	683	difference = self.successResultOf(defer.ensureDeferred(diff_d))
	684
	685	self.assertEqual(difference, {c.event_id})
	686
	687	def test_multiple_unpersisted_chain(self):
	688	# Test getting the auth difference for a simple chain with multiple
	689	# unpersisted events:
	690	#
	691	# Unpersisted \| Persisted
	692	# \|
	693	# D -> C -\|-> B -> A
	694
	695	a = FakeEvent(
	696	id="A", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	697	).to_event([], [])
	698
	699	b = FakeEvent(
	700	id="B", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	701	).to_event([a.event_id], [])
	702
	703	c = FakeEvent(
	704	id="C", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	705	).to_event([b.event_id], [])
	706
	707	d = FakeEvent(
	708	id="D", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	709	).to_event([c.event_id], [])
	710
	711	persisted_events = {a.event_id: a, b.event_id: b}
	712	unpersited_events = {c.event_id: c, d.event_id: d}
	713
	714	state_sets = [
	715	{"a": a.event_id, "b": b.event_id},
	716	{"c": c.event_id, "d": d.event_id},
	717	]
	718
	719	store = TestStateResolutionStore(persisted_events)
	720
	721	diff_d = _get_auth_chain_difference(
	722	ROOM_ID, state_sets, unpersited_events, store
	723	)
	724	difference = self.successResultOf(defer.ensureDeferred(diff_d))
	725
	726	self.assertEqual(difference, {d.event_id, c.event_id})
	727
	728	def test_unpersisted_events_different_sets(self):
	729	# Test getting the auth difference for with multiple unpersisted events
	730	# in different branches:
	731	#
	732	# Unpersisted \| Persisted
	733	# \|
	734	# D --> C -\|-> B -> A
	735	# E ----^ -\|---^
	736	# \|
	737
	738	a = FakeEvent(
	739	id="A", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	740	).to_event([], [])
	741
	742	b = FakeEvent(
	743	id="B", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	744	).to_event([a.event_id], [])
	745
	746	c = FakeEvent(
	747	id="C", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	748	).to_event([b.event_id], [])
	749
	750	d = FakeEvent(
	751	id="D", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	752	).to_event([c.event_id], [])
	753
	754	e = FakeEvent(
	755	id="E", sender=ALICE, type=EventTypes.Member, state_key="", content={},
	756	).to_event([c.event_id, b.event_id], [])
	757
	758	persisted_events = {a.event_id: a, b.event_id: b}
	759	unpersited_events = {c.event_id: c, d.event_id: d, e.event_id: e}
	760
	761	state_sets = [
	762	{"a": a.event_id, "b": b.event_id, "e": e.event_id},
	763	{"c": c.event_id, "d": d.event_id},
	764	]
	765
	766	store = TestStateResolutionStore(persisted_events)
	767
	768	diff_d = _get_auth_chain_difference(
	769	ROOM_ID, state_sets, unpersited_events, store
	770	)
	771	difference = self.successResultOf(defer.ensureDeferred(diff_d))
	772
	773	self.assertEqual(difference, {d.event_id, e.event_id})
	774
	775
589	776	def pairwise(iterable):
590	777	"s -> (s0,s1), (s1,s2), (s2, s3), ..."
591	778	a, b = itertools.tee(iterable)

646	833
647	834	return list(result)
648	835
649		def get_auth_chain_difference(self, auth_sets):
	836	def get_auth_chain_difference(self, room_id, auth_sets):
650	837	chains = [frozenset(self._get_auth_chain(a)) for a in auth_sets]
651	838
652	839	common = set(chains[0]).intersection(*chains[1:])

+26

-0

tests/storage/test_devices.py less more

79	79	)
80	80
81	81	@defer.inlineCallbacks
	82	def test_count_devices_by_users(self):
	83	yield defer.ensureDeferred(
	84	self.store.store_device("user_id", "device1", "display_name 1")
	85	)
	86	yield defer.ensureDeferred(
	87	self.store.store_device("user_id", "device2", "display_name 2")
	88	)
	89	yield defer.ensureDeferred(
	90	self.store.store_device("user_id2", "device3", "display_name 3")
	91	)
	92
	93	res = yield defer.ensureDeferred(self.store.count_devices_by_users())
	94	self.assertEqual(0, res)
	95
	96	res = yield defer.ensureDeferred(self.store.count_devices_by_users(["unknown"]))
	97	self.assertEqual(0, res)
	98
	99	res = yield defer.ensureDeferred(self.store.count_devices_by_users(["user_id"]))
	100	self.assertEqual(2, res)
	101
	102	res = yield defer.ensureDeferred(
	103	self.store.count_devices_by_users(["user_id", "user_id2"])
	104	)
	105	self.assertEqual(3, res)
	106
	107	@defer.inlineCallbacks
82	108	def test_get_device_updates_by_remote(self):
83	109	device_ids = ["device_id1", "device_id2"]
84	110

+1

-1

tests/storage/test_e2e_room_keys.py less more

25	25
26	26	class E2eRoomKeysHandlerTestCase(unittest.HomeserverTestCase):
27	27	def make_homeserver(self, reactor, clock):
28		hs = self.setup_test_homeserver("server", http_client=None)
	28	hs = self.setup_test_homeserver("server", federation_http_client=None)
29	29	self.store = hs.get_datastore()
30	30	return hs
31	31

+14

-7

tests/storage/test_event_federation.py less more

201	201	# Now actually test that various combinations give the right result:
202	202
203	203	difference = self.get_success(
204		self.store.get_auth_chain_difference([{"a"}, {"b"}])
	204	self.store.get_auth_chain_difference(room_id, [{"a"}, {"b"}])
205	205	)
206	206	self.assertSetEqual(difference, {"a", "b"})
207	207
208	208	difference = self.get_success(
209		self.store.get_auth_chain_difference([{"a"}, {"b"}, {"c"}])
	209	self.store.get_auth_chain_difference(room_id, [{"a"}, {"b"}, {"c"}])
210	210	)
211	211	self.assertSetEqual(difference, {"a", "b", "c", "e", "f"})
212	212
213	213	difference = self.get_success(
214		self.store.get_auth_chain_difference([{"a", "c"}, {"b"}])
	214	self.store.get_auth_chain_difference(room_id, [{"a", "c"}, {"b"}])
215	215	)
216	216	self.assertSetEqual(difference, {"a", "b", "c"})
217	217
218	218	difference = self.get_success(
219		self.store.get_auth_chain_difference([{"a"}, {"b"}, {"d"}])
	219	self.store.get_auth_chain_difference(room_id, [{"a", "c"}, {"b", "c"}])
	220	)
	221	self.assertSetEqual(difference, {"a", "b"})
	222
	223	difference = self.get_success(
	224	self.store.get_auth_chain_difference(room_id, [{"a"}, {"b"}, {"d"}])
220	225	)
221	226	self.assertSetEqual(difference, {"a", "b", "d", "e"})
222	227
223	228	difference = self.get_success(
224		self.store.get_auth_chain_difference([{"a"}, {"b"}, {"c"}, {"d"}])
	229	self.store.get_auth_chain_difference(room_id, [{"a"}, {"b"}, {"c"}, {"d"}])
225	230	)
226	231	self.assertSetEqual(difference, {"a", "b", "c", "d", "e", "f"})
227	232
228	233	difference = self.get_success(
229		self.store.get_auth_chain_difference([{"a"}, {"b"}, {"e"}])
	234	self.store.get_auth_chain_difference(room_id, [{"a"}, {"b"}, {"e"}])
230	235	)
231	236	self.assertSetEqual(difference, {"a", "b"})
232	237
233		difference = self.get_success(self.store.get_auth_chain_difference([{"a"}]))
	238	difference = self.get_success(
	239	self.store.get_auth_chain_difference(room_id, [{"a"}])
	240	)
234	241	self.assertSetEqual(difference, set())

+334

-0

tests/storage/test_events.py less more

	0	# -- coding: utf-8 --
	1	# Copyright 2020 The Matrix.org Foundation C.I.C.
	2	#
	3	# Licensed under the Apache License, Version 2.0 (the "License");
	4	# you may not use this file except in compliance with the License.
	5	# You may obtain a copy of the License at
	6	#
	7	# http://www.apache.org/licenses/LICENSE-2.0
	8	#
	9	# Unless required by applicable law or agreed to in writing, software
	10	# distributed under the License is distributed on an "AS IS" BASIS,
	11	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	12	# See the License for the specific language governing permissions and
	13	# limitations under the License.
	14
	15
	16	from synapse.api.constants import EventTypes, Membership
	17	from synapse.api.room_versions import RoomVersions
	18	from synapse.federation.federation_base import event_from_pdu_json
	19	from synapse.rest import admin
	20	from synapse.rest.client.v1 import login, room
	21
	22	from tests.unittest import HomeserverTestCase
	23
	24
	25	class ExtremPruneTestCase(HomeserverTestCase):
	26	servlets = [
	27	admin.register_servlets,
	28	room.register_servlets,
	29	login.register_servlets,
	30	]
	31
	32	def prepare(self, reactor, clock, homeserver):
	33	self.state = self.hs.get_state_handler()
	34	self.persistence = self.hs.get_storage().persistence
	35	self.store = self.hs.get_datastore()
	36
	37	self.register_user("user", "pass")
	38	self.token = self.login("user", "pass")
	39
	40	self.room_id = self.helper.create_room_as(
	41	"user", room_version=RoomVersions.V6.identifier, tok=self.token
	42	)
	43
	44	body = self.helper.send(self.room_id, body="Test", tok=self.token)
	45	local_message_event_id = body["event_id"]
	46
	47	# Fudge a remote event and persist it. This will be the extremity before
	48	# the gap.
	49	self.remote_event_1 = event_from_pdu_json(
	50	{
	51	"type": EventTypes.Message,
	52	"state_key": "@user:other",
	53	"content": {},
	54	"room_id": self.room_id,
	55	"sender": "@user:other",
	56	"depth": 5,
	57	"prev_events": [local_message_event_id],
	58	"auth_events": [],
	59	"origin_server_ts": self.clock.time_msec(),
	60	},
	61	RoomVersions.V6,
	62	)
	63
	64	self.persist_event(self.remote_event_1)
	65
	66	# Check that the current extremities is the remote event.
	67	self.assert_extremities([self.remote_event_1.event_id])
	68
	69	def persist_event(self, event, state=None):
	70	"""Persist the event, with optional state
	71	"""
	72	context = self.get_success(
	73	self.state.compute_event_context(event, old_state=state)
	74	)
	75	self.get_success(self.persistence.persist_event(event, context))
	76
	77	def assert_extremities(self, expected_extremities):
	78	"""Assert the current extremities for the room
	79	"""
	80	extremities = self.get_success(
	81	self.store.get_prev_events_for_room(self.room_id)
	82	)
	83	self.assertCountEqual(extremities, expected_extremities)
	84
	85	def test_prune_gap(self):
	86	"""Test that we drop extremities after a gap when we see an event from
	87	the same domain.
	88	"""
	89
	90	# Fudge a second event which points to an event we don't have. This is a
	91	# state event so that the state changes (otherwise we won't prune the
	92	# extremity as they'll have the same state group).
	93	remote_event_2 = event_from_pdu_json(
	94	{
	95	"type": EventTypes.Member,
	96	"state_key": "@user:other",
	97	"content": {"membership": Membership.JOIN},
	98	"room_id": self.room_id,
	99	"sender": "@user:other",
	100	"depth": 50,
	101	"prev_events": ["$some_unknown_message"],
	102	"auth_events": [],
	103	"origin_server_ts": self.clock.time_msec(),
	104	},
	105	RoomVersions.V6,
	106	)
	107
	108	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	109
	110	self.persist_event(remote_event_2, state=state_before_gap.values())
	111
	112	# Check the new extremity is just the new remote event.
	113	self.assert_extremities([remote_event_2.event_id])
	114
	115	def test_do_not_prune_gap_if_state_different(self):
	116	"""Test that we don't prune extremities after a gap if the resolved
	117	state is different.
	118	"""
	119
	120	# Fudge a second event which points to an event we don't have.
	121	remote_event_2 = event_from_pdu_json(
	122	{
	123	"type": EventTypes.Message,
	124	"state_key": "@user:other",
	125	"content": {},
	126	"room_id": self.room_id,
	127	"sender": "@user:other",
	128	"depth": 10,
	129	"prev_events": ["$some_unknown_message"],
	130	"auth_events": [],
	131	"origin_server_ts": self.clock.time_msec(),
	132	},
	133	RoomVersions.V6,
	134	)
	135
	136	# Now we persist it with state with a dropped history visibility
	137	# setting. The state resolution across the old and new event will then
	138	# include it, and so the resolved state won't match the new state.
	139	state_before_gap = dict(
	140	self.get_success(self.state.get_current_state(self.room_id))
	141	)
	142	state_before_gap.pop(("m.room.history_visibility", ""))
	143
	144	context = self.get_success(
	145	self.state.compute_event_context(
	146	remote_event_2, old_state=state_before_gap.values()
	147	)
	148	)
	149
	150	self.get_success(self.persistence.persist_event(remote_event_2, context))
	151
	152	# Check that we haven't dropped the old extremity.
	153	self.assert_extremities([self.remote_event_1.event_id, remote_event_2.event_id])
	154
	155	def test_prune_gap_if_old(self):
	156	"""Test that we drop extremities after a gap when the previous extremity
	157	is "old"
	158	"""
	159
	160	# Advance the clock for many days to make the old extremity "old". We
	161	# also set the depth to "lots".
	162	self.reactor.advance(7 * 24 * 60 * 60)
	163
	164	# Fudge a second event which points to an event we don't have. This is a
	165	# state event so that the state changes (otherwise we won't prune the
	166	# extremity as they'll have the same state group).
	167	remote_event_2 = event_from_pdu_json(
	168	{
	169	"type": EventTypes.Member,
	170	"state_key": "@user:other2",
	171	"content": {"membership": Membership.JOIN},
	172	"room_id": self.room_id,
	173	"sender": "@user:other2",
	174	"depth": 10000,
	175	"prev_events": ["$some_unknown_message"],
	176	"auth_events": [],
	177	"origin_server_ts": self.clock.time_msec(),
	178	},
	179	RoomVersions.V6,
	180	)
	181
	182	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	183
	184	self.persist_event(remote_event_2, state=state_before_gap.values())
	185
	186	# Check the new extremity is just the new remote event.
	187	self.assert_extremities([remote_event_2.event_id])
	188
	189	def test_do_not_prune_gap_if_other_server(self):
	190	"""Test that we do not drop extremities after a gap when we see an event
	191	from a different domain.
	192	"""
	193
	194	# Fudge a second event which points to an event we don't have. This is a
	195	# state event so that the state changes (otherwise we won't prune the
	196	# extremity as they'll have the same state group).
	197	remote_event_2 = event_from_pdu_json(
	198	{
	199	"type": EventTypes.Member,
	200	"state_key": "@user:other2",
	201	"content": {"membership": Membership.JOIN},
	202	"room_id": self.room_id,
	203	"sender": "@user:other2",
	204	"depth": 10,
	205	"prev_events": ["$some_unknown_message"],
	206	"auth_events": [],
	207	"origin_server_ts": self.clock.time_msec(),
	208	},
	209	RoomVersions.V6,
	210	)
	211
	212	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	213
	214	self.persist_event(remote_event_2, state=state_before_gap.values())
	215
	216	# Check the new extremity is just the new remote event.
	217	self.assert_extremities([self.remote_event_1.event_id, remote_event_2.event_id])
	218
	219	def test_prune_gap_if_dummy_remote(self):
	220	"""Test that we drop extremities after a gap when the previous extremity
	221	is a local dummy event and only points to remote events.
	222	"""
	223
	224	body = self.helper.send_event(
	225	self.room_id, type=EventTypes.Dummy, content={}, tok=self.token
	226	)
	227	local_message_event_id = body["event_id"]
	228	self.assert_extremities([local_message_event_id])
	229
	230	# Advance the clock for many days to make the old extremity "old". We
	231	# also set the depth to "lots".
	232	self.reactor.advance(7 * 24 * 60 * 60)
	233
	234	# Fudge a second event which points to an event we don't have. This is a
	235	# state event so that the state changes (otherwise we won't prune the
	236	# extremity as they'll have the same state group).
	237	remote_event_2 = event_from_pdu_json(
	238	{
	239	"type": EventTypes.Member,
	240	"state_key": "@user:other2",
	241	"content": {"membership": Membership.JOIN},
	242	"room_id": self.room_id,
	243	"sender": "@user:other2",
	244	"depth": 10000,
	245	"prev_events": ["$some_unknown_message"],
	246	"auth_events": [],
	247	"origin_server_ts": self.clock.time_msec(),
	248	},
	249	RoomVersions.V6,
	250	)
	251
	252	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	253
	254	self.persist_event(remote_event_2, state=state_before_gap.values())
	255
	256	# Check the new extremity is just the new remote event.
	257	self.assert_extremities([remote_event_2.event_id])
	258
	259	def test_prune_gap_if_dummy_local(self):
	260	"""Test that we don't drop extremities after a gap when the previous
	261	extremity is a local dummy event and points to local events.
	262	"""
	263
	264	body = self.helper.send(self.room_id, body="Test", tok=self.token)
	265
	266	body = self.helper.send_event(
	267	self.room_id, type=EventTypes.Dummy, content={}, tok=self.token
	268	)
	269	local_message_event_id = body["event_id"]
	270	self.assert_extremities([local_message_event_id])
	271
	272	# Advance the clock for many days to make the old extremity "old". We
	273	# also set the depth to "lots".
	274	self.reactor.advance(7 * 24 * 60 * 60)
	275
	276	# Fudge a second event which points to an event we don't have. This is a
	277	# state event so that the state changes (otherwise we won't prune the
	278	# extremity as they'll have the same state group).
	279	remote_event_2 = event_from_pdu_json(
	280	{
	281	"type": EventTypes.Member,
	282	"state_key": "@user:other2",
	283	"content": {"membership": Membership.JOIN},
	284	"room_id": self.room_id,
	285	"sender": "@user:other2",
	286	"depth": 10000,
	287	"prev_events": ["$some_unknown_message"],
	288	"auth_events": [],
	289	"origin_server_ts": self.clock.time_msec(),
	290	},
	291	RoomVersions.V6,
	292	)
	293
	294	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	295
	296	self.persist_event(remote_event_2, state=state_before_gap.values())
	297
	298	# Check the new extremity is just the new remote event.
	299	self.assert_extremities([remote_event_2.event_id, local_message_event_id])
	300
	301	def test_do_not_prune_gap_if_not_dummy(self):
	302	"""Test that we do not drop extremities after a gap when the previous extremity
	303	is not a dummy event.
	304	"""
	305
	306	body = self.helper.send(self.room_id, body="test", tok=self.token)
	307	local_message_event_id = body["event_id"]
	308	self.assert_extremities([local_message_event_id])
	309
	310	# Fudge a second event which points to an event we don't have. This is a
	311	# state event so that the state changes (otherwise we won't prune the
	312	# extremity as they'll have the same state group).
	313	remote_event_2 = event_from_pdu_json(
	314	{
	315	"type": EventTypes.Member,
	316	"state_key": "@user:other2",
	317	"content": {"membership": Membership.JOIN},
	318	"room_id": self.room_id,
	319	"sender": "@user:other2",
	320	"depth": 10000,
	321	"prev_events": ["$some_unknown_message"],
	322	"auth_events": [],
	323	"origin_server_ts": self.clock.time_msec(),
	324	},
	325	RoomVersions.V6,
	326	)
	327
	328	state_before_gap = self.get_success(self.state.get_current_state(self.room_id))
	329
	330	self.persist_event(remote_event_2, state=state_before_gap.values())
	331
	332	# Check the new extremity is just the new remote event.
	333	self.assert_extremities([local_message_event_id, remote_event_2.event_id])

+7

-0

tests/storage/test_main.py less more

47	47
48	48	self.assertEquals(1, total)
49	49	self.assertEquals(self.displayname, users.pop()["displayname"])
	50
	51	users, total = yield defer.ensureDeferred(
	52	self.store.get_users_paginate(0, 10, name="BC", guests=False)
	53	)
	54
	55	self.assertEquals(1, total)
	56	self.assertEquals(self.displayname, users.pop()["displayname"])

+1

-1

tests/storage/test_purge.py less more

26	26	servlets = [room.register_servlets]
27	27
28	28	def make_homeserver(self, reactor, clock):
29		hs = self.setup_test_homeserver("server", http_client=None)
	29	hs = self.setup_test_homeserver("server", federation_http_client=None)
30	30	return hs
31	31
32	32	def prepare(self, reactor, clock, hs):

+3

-8

tests/storage/test_redaction.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
16
17		from mock import Mock
18
19	16	from canonicaljson import json
20	17
21	18	from twisted.internet import defer

29	26
30	27
31	28	class RedactionTestCase(unittest.HomeserverTestCase):
32		def make_homeserver(self, reactor, clock):
33		config = self.default_config()
	29	def default_config(self):
	30	config = super().default_config()
34	31	config["redaction_retention_period"] = "30d"
35		return self.setup_test_homeserver(
36		resource_for_federation=Mock(), http_client=None, config=config
37		)
	32	return config
38	33
39	34	def prepare(self, reactor, clock, hs):
40	35	self.store = hs.get_datastore()

+0

-8

tests/storage/test_roommember.py less more

13	13	# See the License for the specific language governing permissions and
14	14	# limitations under the License.
15	15
16		from unittest.mock import Mock
17
18	16	from synapse.api.constants import Membership
19	17	from synapse.rest.admin import register_servlets_for_client_rest_resource
20	18	from synapse.rest.client.v1 import login, room

32	30	register_servlets_for_client_rest_resource,
33	31	room.register_servlets,
34	32	]
35
36		def make_homeserver(self, reactor, clock):
37		hs = self.setup_test_homeserver(
38		resource_for_federation=Mock(), http_client=None
39		)
40		return hs
41	33
42	34	def prepare(self, reactor, clock, hs: TestHomeServer):
43	35

+23

-0

tests/storage/test_user_directory.py less more

20	20	ALICE = "@alice:a"
21	21	BOB = "@bob:b"
22	22	BOBBY = "@bobby:a"
	23	# The localpart isn't 'Bela' on purpose so we can test looking up display names.
	24	BELA = "@somenickname:a"
23	25
24	26
25	27	class UserDirectoryStoreTestCase(unittest.TestCase):

38	40	)
39	41	yield defer.ensureDeferred(
40	42	self.store.update_profile_in_user_dir(BOBBY, "bobby", None)
	43	)
	44	yield defer.ensureDeferred(
	45	self.store.update_profile_in_user_dir(BELA, "Bela", None)
41	46	)
42	47	yield defer.ensureDeferred(
43	48	self.store.add_users_in_public_rooms("!room:id", (ALICE, BOB))

71	76	)
72	77	finally:
73	78	self.hs.config.user_directory_search_all_users = False
	79
	80	@defer.inlineCallbacks
	81	def test_search_user_dir_stop_words(self):
	82	"""Tests that a user can look up another user by searching for the start if its
	83	display name even if that name happens to be a common English word that would
	84	usually be ignored in full text searches.
	85	"""
	86	self.hs.config.user_directory_search_all_users = True
	87	try:
	88	r = yield defer.ensureDeferred(self.store.search_user_dir(ALICE, "be", 10))
	89	self.assertFalse(r["limited"])
	90	self.assertEqual(1, len(r["results"]))
	91	self.assertDictEqual(
	92	r["results"][0],
	93	{"user_id": BELA, "display_name": "Bela", "avatar_url": None},
	94	)
	95	finally:
	96	self.hs.config.user_directory_search_all_users = False

+2

-2

tests/test_federation.py less more

36	36	self.hs_clock = Clock(self.reactor)
37	37	self.homeserver = setup_test_homeserver(
38	38	self.addCleanup,
39		http_client=self.http_client,
	39	federation_http_client=self.http_client,
40	40	clock=self.hs_clock,
41	41	reactor=self.reactor,
42	42	)

133	133	}
134	134	)
135	135
136		with LoggingContext(request="lying_event"):
	136	with LoggingContext():
137	137	failure = self.get_failure(
138	138	self.handler.on_receive_pdu(
139	139	"test.serv", lying_event, sent_to_us_directly=True

+45

-3

tests/test_mau.py less more

18	18
19	19	from synapse.api.constants import LoginType
20	20	from synapse.api.errors import Codes, HttpResponseException, SynapseError
	21	from synapse.appservice import ApplicationService
21	22	from synapse.rest.client.v2_alpha import register, sync
22	23
23	24	from tests import unittest

74	75	self.assertEqual(e.code, 403)
75	76	self.assertEqual(e.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
76	77
	78	def test_as_ignores_mau(self):
	79	"""Test that application services can still create users when the MAU
	80	limit has been reached. This only works when application service
	81	user ip tracking is disabled.
	82	"""
	83
	84	# Create and sync so that the MAU counts get updated
	85	token1 = self.create_user("kermit1")
	86	self.do_sync_for_user(token1)
	87	token2 = self.create_user("kermit2")
	88	self.do_sync_for_user(token2)
	89
	90	# check we're testing what we think we are: there should be two active users
	91	self.assertEqual(self.get_success(self.store.get_monthly_active_count()), 2)
	92
	93	# We've created and activated two users, we shouldn't be able to
	94	# register new users
	95	with self.assertRaises(SynapseError) as cm:
	96	self.create_user("kermit3")
	97
	98	e = cm.exception
	99	self.assertEqual(e.code, 403)
	100	self.assertEqual(e.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
	101
	102	# Cheekily add an application service that we use to register a new user
	103	# with.
	104	as_token = "foobartoken"
	105	self.store.services_cache.append(
	106	ApplicationService(
	107	token=as_token,
	108	hostname=self.hs.hostname,
	109	id="SomeASID",
	110	sender="@as_sender:test",
	111	namespaces={"users": [{"regex": "@as_*", "exclusive": True}]},
	112	)
	113	)
	114
	115	self.create_user("as_kermit4", token=as_token)
	116
77	117	def test_allowed_after_a_month_mau(self):
78	118	# Create and sync so that the MAU counts get updated
79	119	token1 = self.create_user("kermit1")

191	231	self.reactor.advance(100)
192	232	self.assertEqual(2, self.successResultOf(count))
193	233
194		def create_user(self, localpart):
	234	def create_user(self, localpart, token=None):
195	235	request_data = json.dumps(
196	236	{
197	237	"username": localpart,

200	240	}
201	241	)
202	242
203		request, channel = self.make_request("POST", "/register", request_data)
	243	channel = self.make_request(
	244	"POST", "/register", request_data, access_token=token,
	245	)
204	246
205	247	if channel.code != 200:
206	248	raise HttpResponseException(

212	254	return access_token
213	255
214	256	def do_sync_for_user(self, token):
215		request, channel = self.make_request("GET", "/sync", access_token=token)
	257	channel = self.make_request("GET", "/sync", access_token=token)
216	258
217	259	if channel.code != 200:
218	260	raise HttpResponseException(

+16

-11

tests/test_preview.py less more

55	55
56	56	desc = summarize_paragraphs(example_paras, min_size=200, max_size=500)
57	57
58		self.assertEquals(
	58	self.assertEqual(
59	59	desc,
60	60	"Tromsø (Norwegian pronunciation: [ˈtrʊmsœ] ( listen); Northern Sami:"
61	61	" Romsa; Finnish: Tromssa[2] Kven: Tromssa) is a city and municipality in"

68	68
69	69	desc = summarize_paragraphs(example_paras[1:], min_size=200, max_size=500)
70	70
71		self.assertEquals(
	71	self.assertEqual(
72	72	desc,
73	73	"Tromsø lies in Northern Norway. The municipality has a population of"
74	74	" (2015) 72,066, but with an annual influx of students it has over 75,000"

95	95
96	96	desc = summarize_paragraphs(example_paras, min_size=200, max_size=500)
97	97
98		self.assertEquals(
	98	self.assertEqual(
99	99	desc,
100	100	"Tromsø (Norwegian pronunciation: [ˈtrʊmsœ] ( listen); Northern Sami:"
101	101	" Romsa; Finnish: Tromssa[2] Kven: Tromssa) is a city and municipality in"

121	121	]
122	122
123	123	desc = summarize_paragraphs(example_paras, min_size=200, max_size=500)
124		self.assertEquals(
	124	self.assertEqual(
125	125	desc,
126	126	"Tromsø (Norwegian pronunciation: [ˈtrʊmsœ] ( listen); Northern Sami:"
127	127	" Romsa; Finnish: Tromssa[2] Kven: Tromssa) is a city and municipality in"

148	148
149	149	og = decode_and_calc_og(html, "http://example.com/test.html")
150	150
151		self.assertEquals(og, {"og:title": "Foo", "og:description": "Some text."})
	151	self.assertEqual(og, {"og:title": "Foo", "og:description": "Some text."})
152	152
153	153	def test_comment(self):
154	154	html = """

163	163
164	164	og = decode_and_calc_og(html, "http://example.com/test.html")
165	165
166		self.assertEquals(og, {"og:title": "Foo", "og:description": "Some text."})
	166	self.assertEqual(og, {"og:title": "Foo", "og:description": "Some text."})
167	167
168	168	def test_comment2(self):
169	169	html = """

181	181
182	182	og = decode_and_calc_og(html, "http://example.com/test.html")
183	183
184		self.assertEquals(
	184	self.assertEqual(
185	185	og,
186	186	{
187	187	"og:title": "Foo",

202	202
203	203	og = decode_and_calc_og(html, "http://example.com/test.html")
204	204
205		self.assertEquals(og, {"og:title": "Foo", "og:description": "Some text."})
	205	self.assertEqual(og, {"og:title": "Foo", "og:description": "Some text."})
206	206
207	207	def test_missing_title(self):
208	208	html = """

215	215
216	216	og = decode_and_calc_og(html, "http://example.com/test.html")
217	217
218		self.assertEquals(og, {"og:title": None, "og:description": "Some text."})
	218	self.assertEqual(og, {"og:title": None, "og:description": "Some text."})
219	219
220	220	def test_h1_as_title(self):
221	221	html = """

229	229
230	230	og = decode_and_calc_og(html, "http://example.com/test.html")
231	231
232		self.assertEquals(og, {"og:title": "Title", "og:description": "Some text."})
	232	self.assertEqual(og, {"og:title": "Title", "og:description": "Some text."})
233	233
234	234	def test_missing_title_and_broken_h1(self):
235	235	html = """

243	243
244	244	og = decode_and_calc_og(html, "http://example.com/test.html")
245	245
246		self.assertEquals(og, {"og:title": None, "og:description": "Some text."})
	246	self.assertEqual(og, {"og:title": None, "og:description": "Some text."})
	247
	248	def test_empty(self):
	249	html = ""
	250	og = decode_and_calc_og(html, "http://example.com/test.html")
	251	self.assertEqual(og, {})

+15

-15

tests/test_server.py less more

37	37	self.reactor = ThreadedMemoryReactorClock()
38	38	self.hs_clock = Clock(self.reactor)
39	39	self.homeserver = setup_test_homeserver(
40		self.addCleanup, http_client=None, clock=self.hs_clock, reactor=self.reactor
	40	self.addCleanup,
	41	federation_http_client=None,
	42	clock=self.hs_clock,
	43	reactor=self.reactor,
41	44	)
42	45
43	46	def test_handler_for_request(self):

60	63	"test_servlet",
61	64	)
62	65
63		request, channel = make_request(
	66	make_request(
64	67	self.reactor, FakeSite(res), b"GET", b"/_matrix/foo/%E2%98%83?a=%E2%98%83"
65	68	)
66	69
67		self.assertEqual(request.args, {b"a": ["\N{SNOWMAN}".encode("utf8")]})
68	70	self.assertEqual(got_kwargs, {"room_id": "\N{SNOWMAN}"})
69	71
70	72	def test_callback_direct_exception(self):

81	83	"GET", [re.compile("^/_matrix/foo$")], _callback, "test_servlet"
82	84	)
83	85
84		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
	86	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
85	87
86	88	self.assertEqual(channel.result["code"], b"500")
87	89

105	107	"GET", [re.compile("^/_matrix/foo$")], _callback, "test_servlet"
106	108	)
107	109
108		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
	110	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
109	111
110	112	self.assertEqual(channel.result["code"], b"500")
111	113

123	125	"GET", [re.compile("^/_matrix/foo$")], _callback, "test_servlet"
124	126	)
125	127
126		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
	128	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foo")
127	129
128	130	self.assertEqual(channel.result["code"], b"403")
129	131	self.assertEqual(channel.json_body["error"], "Forbidden!!one!")

145	147	"GET", [re.compile("^/_matrix/foo$")], _callback, "test_servlet"
146	148	)
147	149
148		_, channel = make_request(
149		self.reactor, FakeSite(res), b"GET", b"/_matrix/foobar"
150		)
	150	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/_matrix/foobar")
151	151
152	152	self.assertEqual(channel.result["code"], b"400")
153	153	self.assertEqual(channel.json_body["error"], "Unrecognized request")

169	169	)
170	170
171	171	# The path was registered as GET, but this is a HEAD request.
172		_, channel = make_request(self.reactor, FakeSite(res), b"HEAD", b"/_matrix/foo")
	172	channel = make_request(self.reactor, FakeSite(res), b"HEAD", b"/_matrix/foo")
173	173
174	174	self.assertEqual(channel.result["code"], b"200")
175	175	self.assertNotIn("body", channel.result)

201	201	)
202	202
203	203	# render the request and return the channel
204		_, channel = make_request(self.reactor, site, method, path, shorthand=False)
	204	channel = make_request(self.reactor, site, method, path, shorthand=False)
205	205	return channel
206	206
207	207	def test_unknown_options_request(self):

274	274	res = WrapHtmlRequestHandlerTests.TestResource()
275	275	res.callback = callback
276	276
277		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
	277	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
278	278
279	279	self.assertEqual(channel.result["code"], b"200")
280	280	body = channel.result["body"]

292	292	res = WrapHtmlRequestHandlerTests.TestResource()
293	293	res.callback = callback
294	294
295		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
	295	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
296	296
297	297	self.assertEqual(channel.result["code"], b"301")
298	298	headers = channel.result["headers"]

313	313	res = WrapHtmlRequestHandlerTests.TestResource()
314	314	res.callback = callback
315	315
316		_, channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
	316	channel = make_request(self.reactor, FakeSite(res), b"GET", b"/path")
317	317
318	318	self.assertEqual(channel.result["code"], b"304")
319	319	headers = channel.result["headers"]

332	332	res = WrapHtmlRequestHandlerTests.TestResource()
333	333	res.callback = callback
334	334
335		_, channel = make_request(self.reactor, FakeSite(res), b"HEAD", b"/path")
	335	channel = make_request(self.reactor, FakeSite(res), b"HEAD", b"/path")
336	336
337	337	self.assertEqual(channel.result["code"], b"200")
338	338	self.assertNotIn("body", channel.result)

+3

-3

tests/test_terms_auth.py less more

52	52	def test_ui_auth(self):
53	53	# Do a UI auth request
54	54	request_data = json.dumps({"username": "kermit", "password": "monkey"})
55		request, channel = self.make_request(b"POST", self.url, request_data)
	55	channel = self.make_request(b"POST", self.url, request_data)
56	56
57	57	self.assertEquals(channel.result["code"], b"401", channel.result)
58	58

95	95
96	96	self.registration_handler.check_username = Mock(return_value=True)
97	97
98		request, channel = self.make_request(b"POST", self.url, request_data)
	98	channel = self.make_request(b"POST", self.url, request_data)
99	99
100	100	# We don't bother checking that the response is correct - we'll leave that to
101	101	# other tests. We just want to make sure we're on the right path.

112	112	},
113	113	}
114	114	)
115		request, channel = self.make_request(b"POST", self.url, request_data)
	115	channel = self.make_request(b"POST", self.url, request_data)
116	116
117	117	# We're interested in getting a response that looks like a successful
118	118	# registration, not so much that the details are exactly what we want.

+39

-0

tests/test_utils/__init__.py less more

20	20	import warnings
21	21	from asyncio import Future
22	22	from typing import Any, Awaitable, Callable, TypeVar
	23
	24	from mock import Mock
	25
	26	import attr
	27
	28	from twisted.python.failure import Failure
	29	from twisted.web.client import ResponseDone
23	30
24	31	TV = TypeVar("TV")
25	32

79	86	sys.unraisablehook = unraisablehook # type: ignore
80	87
81	88	return cleanup
	89
	90
	91	def simple_async_mock(return_value=None, raises=None) -> Mock:
	92	# AsyncMock is not available in python3.5, this mimics part of its behaviour
	93	async def cb(args, *kwargs):
	94	if raises:
	95	raise raises
	96	return return_value
	97
	98	return Mock(side_effect=cb)
	99
	100
	101	@attr.s
	102	class FakeResponse:
	103	"""A fake twisted.web.IResponse object
	104
	105	there is a similar class at treq.test.test_response, but it lacks a `phrase`
	106	attribute, and didn't support deliverBody until recently.
	107	"""
	108
	109	# HTTP response code
	110	code = attr.ib(type=int)
	111
	112	# HTTP response phrase (eg b'OK' for a 200)
	113	phrase = attr.ib(type=bytes)
	114
	115	# body of the response
	116	body = attr.ib(type=bytes)
	117
	118	def deliverBody(self, protocol):
	119	protocol.dataReceived(self.body)
	120	protocol.connectionLost(Failure(ResponseDone()))

+1

-1

tests/test_utils/logging_setup.py less more

47	47	handler = ToTwistedHandler()
48	48	formatter = logging.Formatter(log_format)
49	49	handler.setFormatter(formatter)
50		handler.addFilter(LoggingContextFilter(request=""))
	50	handler.addFilter(LoggingContextFilter())
51	51	root_logger.addHandler(handler)
52	52
53	53	log_level = os.environ.get("SYNAPSE_TEST_LOG_LEVEL", "ERROR")

+54

-53

tests/unittest.py less more

19	19	import inspect
20	20	import logging
21	21	import time
22		from typing import Optional, Tuple, Type, TypeVar, Union, overload
	22	from typing import Dict, Iterable, Optional, Tuple, Type, TypeVar, Union
23	23
24	24	from mock import Mock, patch
25	25

45	45	)
46	46	from synapse.server import HomeServer
47	47	from synapse.types import UserID, create_requester
	48	from synapse.util.httpresourcetree import create_resource_tree
48	49	from synapse.util.ratelimitutils import FederationRateLimiter
49	50
50	51	from tests.server import FakeChannel, get_clock, make_request, setup_test_homeserver

319	320	"""
320	321	Create a the root resource for the test server.
321	322
	323	The default calls `self.create_resource_dict` and builds the resultant dict
	324	into a tree.
	325	"""
	326	root_resource = Resource()
	327	create_resource_tree(self.create_resource_dict(), root_resource)
	328	return root_resource
	329
	330	def create_resource_dict(self) -> Dict[str, Resource]:
	331	"""Create a resource tree for the test server
	332
	333	A resource tree is a mapping from path to twisted.web.resource.
	334
322	335	The default implementation creates a JsonResource and calls each function in
323		`servlets` to register servletes against it
324		"""
325		resource = JsonResource(self.hs)
326
	336	`servlets` to register servlets against it.
	337	"""
	338	servlet_resource = JsonResource(self.hs)
327	339	for servlet in self.servlets:
328		servlet(self.hs, resource)
329
330		return resource
	340	servlet(self.hs, servlet_resource)
	341	return {
	342	"/_matrix/client": servlet_resource,
	343	"/_synapse/admin": servlet_resource,
	344	}
331	345
332	346	def default_config(self):
333	347	"""

357	371	Function to optionally be overridden in subclasses.
358	372	"""
359	373
360		# Annoyingly mypy doesn't seem to pick up the fact that T is SynapseRequest
361		# when the `request` arg isn't given, so we define an explicit override to
362		# cover that case.
363		@overload
364		def make_request(
365		self,
366		method: Union[bytes, str],
367		path: Union[bytes, str],
368		content: Union[bytes, dict] = b"",
369		access_token: Optional[str] = None,
370		shorthand: bool = True,
371		federation_auth_origin: str = None,
372		content_is_form: bool = False,
373		await_result: bool = True,
374		) -> Tuple[SynapseRequest, FakeChannel]:
375		...
376
377		@overload
378	374	def make_request(
379	375	self,
380	376	method: Union[bytes, str],

386	382	federation_auth_origin: str = None,
387	383	content_is_form: bool = False,
388	384	await_result: bool = True,
389		) -> Tuple[T, FakeChannel]:
390		...
391
392		def make_request(
393		self,
394		method: Union[bytes, str],
395		path: Union[bytes, str],
396		content: Union[bytes, dict] = b"",
397		access_token: Optional[str] = None,
398		request: Type[T] = SynapseRequest,
399		shorthand: bool = True,
400		federation_auth_origin: str = None,
401		content_is_form: bool = False,
402		await_result: bool = True,
403		) -> Tuple[T, FakeChannel]:
	385	custom_headers: Optional[
	386	Iterable[Tuple[Union[bytes, str], Union[bytes, str]]]
	387	] = None,
	388	) -> FakeChannel:
404	389	"""
405	390	Create a SynapseRequest at the path using the method and containing the
406	391	given content.

422	407	true (the default), will pump the test reactor until the the renderer
423	408	tells the channel the request is finished.
424	409
	410	custom_headers: (name, value) pairs to add as request headers
	411
425	412	Returns:
426		Tuple[synapse.http.site.SynapseRequest, channel]
	413	The FakeChannel object which stores the result of the request.
427	414	"""
428	415	return make_request(
429	416	self.reactor,

437	424	federation_auth_origin,
438	425	content_is_form,
439	426	await_result,
	427	custom_headers,
440	428	)
441	429
442	430	def setup_test_homeserver(self, args, *kwargs):

553	541	self.hs.config.registration_shared_secret = "shared"
554	542
555	543	# Create the user
556		request, channel = self.make_request("GET", "/_synapse/admin/v1/register")
	544	channel = self.make_request("GET", "/_synapse/admin/v1/register")
557	545	self.assertEqual(channel.code, 200, msg=channel.result)
558	546	nonce = channel.json_body["nonce"]
559	547

578	566	"inhibit_login": True,
579	567	}
580	568	)
581		request, channel = self.make_request(
	569	channel = self.make_request(
582	570	"POST", "/_synapse/admin/v1/register", body.encode("utf8")
583	571	)
584	572	self.assertEqual(channel.code, 200, channel.json_body)

596	584	if device_id:
597	585	body["device_id"] = device_id
598	586
599		request, channel = self.make_request(
	587	channel = self.make_request(
600	588	"POST", "/_matrix/client/r0/login", json.dumps(body).encode("utf8")
601	589	)
602	590	self.assertEqual(channel.code, 200, channel.result)

664	652	"""
665	653	body = {"type": "m.login.password", "user": username, "password": password}
666	654
667		request, channel = self.make_request(
	655	channel = self.make_request(
668	656	"POST", "/_matrix/client/r0/login", json.dumps(body).encode("utf8")
669	657	)
670	658	self.assertEqual(channel.code, 403, channel.result)

690	678	A federating homeserver that authenticates incoming requests as `other.example.com`.
691	679	"""
692	680
693		def prepare(self, reactor, clock, homeserver):
	681	def create_resource_dict(self) -> Dict[str, Resource]:
	682	d = super().create_resource_dict()
	683	d["/_matrix/federation"] = TestTransportLayerServer(self.hs)
	684	return d
	685
	686
	687	class TestTransportLayerServer(JsonResource):
	688	"""A test implementation of TransportLayerServer
	689
	690	authenticates incoming requests as `other.example.com`.
	691	"""
	692
	693	def __init__(self, hs):
	694	super().__init__(hs)
	695
694	696	class Authenticator:
695	697	def authenticate_request(self, request, content):
696	698	return succeed("other.example.com")
697	699
	700	authenticator = Authenticator()
	701
698	702	ratelimiter = FederationRateLimiter(
699		clock,
	703	hs.get_clock(),
700	704	FederationRateLimitConfig(
701	705	window_size=1,
702	706	sleep_limit=1,

705	709	concurrent_requests=1000,
706	710	),
707	711	)
708		federation_server.register_servlets(
709		homeserver, self.resource, Authenticator(), ratelimiter
710		)
711
712		return super().prepare(reactor, clock, homeserver)
	712
	713	federation_server.register_servlets(hs, self, authenticator, ratelimiter)
713	714
714	715
715	716	def override_config(extra_config):

+1

-107

tests/utils.py less more

19	19	import time
20	20	import uuid
21	21	import warnings
22		from inspect import getcallargs
23	22	from typing import Type
24	23	from urllib import parse as urlparse
25	24
26	25	from mock import Mock, patch
27	26
28		from twisted.internet import defer, reactor
	27	from twisted.internet import defer
29	28
30	29	from synapse.api.constants import EventTypes
31	30	from synapse.api.errors import CodeMessageException, cs_error

33	32	from synapse.config.database import DatabaseConnectionConfig
34	33	from synapse.config.homeserver import HomeServerConfig
35	34	from synapse.config.server import DEFAULT_ROOM_VERSION
36		from synapse.federation.transport import server as federation_server
37	35	from synapse.http.server import HttpServer
38	36	from synapse.logging.context import current_context, set_current_context
39	37	from synapse.server import HomeServer

41	39	from synapse.storage.database import LoggingDatabaseConnection
42	40	from synapse.storage.engines import PostgresEngine, create_engine
43	41	from synapse.storage.prepare_database import prepare_database
44		from synapse.util.ratelimitutils import FederationRateLimiter
45	42
46	43	# set this to True to run the tests against postgres instead of sqlite.
47	44	#

341	338
342	339	hs.get_auth_handler().validate_hash = validate_hash
343	340
344		fed = kwargs.get("resource_for_federation", None)
345		if fed:
346		register_federation_servlets(hs, fed)
347
348	341	return hs
349
350
351		def register_federation_servlets(hs, resource):
352		federation_server.register_servlets(
353		hs,
354		resource=resource,
355		authenticator=federation_server.Authenticator(hs),
356		ratelimiter=FederationRateLimiter(
357		hs.get_clock(), config=hs.config.rc_federation
358		),
359		)
360
361
362		def get_mock_call_args(pattern_func, mock_func):
363		""" Return the arguments the mock function was called with interpreted
364		by the pattern functions argument list.
365		"""
366		invoked_args, invoked_kargs = mock_func.call_args
367		return getcallargs(pattern_func, invoked_args, *invoked_kargs)
368	342
369	343
370	344	def mock_getRawHeaders(headers=None):

552	526	return d
553	527
554	528
555		def _format_call(args, kwargs):
556		return ", ".join(
557		["%r" % (a) for a in args] + ["%s=%r" % (k, v) for k, v in kwargs.items()]
558		)
559
560
561		class DeferredMockCallable:
562		"""A callable instance that stores a set of pending call expectations and
563		return values for them. It allows a unit test to assert that the given set
564		of function calls are eventually made, by awaiting on them to be called.
565		"""
566
567		def __init__(self):
568		self.expectations = []
569		self.calls = []
570
571		def __call__(self, args, *kwargs):
572		self.calls.append((args, kwargs))
573
574		if not self.expectations:
575		raise ValueError(
576		"%r has no pending calls to handle call(%s)"
577		% (self, _format_call(args, kwargs))
578		)
579
580		for (call, result, d) in self.expectations:
581		if args == call[1] and kwargs == call[2]:
582		d.callback(None)
583		return result
584
585		failure = AssertionError(
586		"Was not expecting call(%s)" % (_format_call(args, kwargs))
587		)
588
589		for _, _, d in self.expectations:
590		try:
591		d.errback(failure)
592		except Exception:
593		pass
594
595		raise failure
596
597		def expect_call_and_return(self, call, result):
598		self.expectations.append((call, result, defer.Deferred()))
599
600		@defer.inlineCallbacks
601		def await_calls(self, timeout=1000):
602		deferred = defer.DeferredList(
603		[d for _, _, d in self.expectations], fireOnOneErrback=True
604		)
605
606		timer = reactor.callLater(
607		timeout / 1000,
608		deferred.errback,
609		AssertionError(
610		"%d pending calls left: %s"
611		% (
612		len([e for e in self.expectations if not e[2].called]),
613		[e for e in self.expectations if not e[2].called],
614		)
615		),
616		)
617
618		yield deferred
619
620		timer.cancel()
621
622		self.calls = []
623
624		def assert_had_no_calls(self):
625		if self.calls:
626		calls = self.calls
627		self.calls = []
628
629		raise AssertionError(
630		"Expected not to received any calls, got:\n"
631		+ "\n".join(["call(%s)" % _format_call(c[0], c[1]) for c in calls])
632		)
633
634
635	529	async def create_room(hs, room_id: str, creator_id: str):
636	530	"""Creates and persist a creation event for the given room
637	531	"""

+29

-20

tox.ini less more

6	6	python-subunit
7	7	junitxml
8	8	coverage
9		coverage-enable-subprocess
	9
	10	# this is pinned since it's a bit of an obscure package.
	11	coverage-enable-subprocess==1.0
10	12
11	13	# cyptography 2.2 requires setuptools >= 18.5
12	14	#

22	24	# install the "enum34" dependency of cryptography.
23	25	pip>=10
24	26
25		setenv =
26		PYTHONDONTWRITEBYTECODE = no_byte_code
27		COVERAGE_PROCESS_START = {toxinidir}/.coveragerc
28
29	27	[testenv]
30	28	deps =
31	29	{[base]deps}
32	30	extras = all, test
33	31
34		whitelist_externals =
35		sh
	32	setenv =
	33	# use a postgres db for tox environments with "-postgres" in the name
	34	# (see https://tox.readthedocs.io/en/3.20.1/config.html#factors-and-factor-conditional-settings)
	35	postgres: SYNAPSE_POSTGRES = 1
36	36
37		setenv =
38		{[base]setenv}
39		postgres: SYNAPSE_POSTGRES = 1
	37	# this is used by .coveragerc to refer to the top of our tree.
40	38	TOP={toxinidir}
41	39
42	40	passenv = *
43	41
44	42	commands =
45		/usr/bin/find "{toxinidir}" -name '*.pyc' -delete
46		# Add this so that coverage will run on subprocesses
47		{envbindir}/coverage run "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
	43	# the "env" invocation enables coverage checking for sub-processes. This is
	44	# particularly important when running trial with `-j`, since that will make
	45	# it run tests in a subprocess, whose coverage would otherwise not be
	46	# tracked. (It also makes an explicit `coverage run` command redundant.)
	47	#
	48	# (See https://coverage.readthedocs.io/en/coverage-5.3/subprocess.html.
	49	# Note that the `coverage.process_startup()` call is done by
	50	# `coverage-enable-subprocess`.)
	51	#
	52	# we use "env" rather than putting a value in `setenv` so that it is not
	53	# inherited by other tox environments.
	54	#
	55	# keep this in sync with the copy in `testenv:py35-old`.
	56	#
	57	/usr/bin/env COVERAGE_PROCESS_START={toxinidir}/.coveragerc "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
48	58
49	59	# As of twisted 16.4, trial tries to import the tests as a package (previously
50	60	# it loaded the files explicitly), which means they need to be on the

79	89
80	90	lxml
81	91	coverage
82		coverage-enable-subprocess
	92	coverage-enable-subprocess==1.0
83	93
84	94	commands =
85		/usr/bin/find "{toxinidir}" -name '*.pyc' -delete
86	95	# Make all greater-thans equals so we test the oldest version of our direct
87	96	# dependencies, but make the pyopenssl 17.0, which can work against an
88	97	# OpenSSL 1.1 compiled cryptography (as older ones don't compile on Travis).

91	100	# Install Synapse itself. This won't update any libraries.
92	101	pip install -e ".[test]"
93	102
94		{envbindir}/coverage run "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
	103	# we have to duplicate the command from `testenv` rather than refer to it
	104	# as `{[testenv]commands}`, because we run on ubuntu xenial, which has
	105	# tox 2.3.1, and https://github.com/tox-dev/tox/issues/208.
	106	#
	107	/usr/bin/env COVERAGE_PROCESS_START={toxinidir}/.coveragerc "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
95	108
96	109	[testenv:benchmark]
97	110	deps =

156	169	{[base]deps}
157	170	extras = all,mypy
158	171	commands = mypy
159
160		# To find all folders that pass mypy you run:
161		#
162		# find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null" \; -print