Codebase list matrix-synapse / 790aead
New upstream version 1.5.0 Andrej Shadura 4 years ago
19 changed file(s) with 243 addition(s) and 209 deletion(s). Raw diff Collapse all Expand all
0 Erik Johnston <erik at matrix.org>
1 * HS core
2 * Federation API impl
0 The following is an incomplete list of people outside the core team who have
1 contributed to Synapse. It is no longer maintained: more recent contributions
2 are listed in the `changelog <CHANGES.md>`_.
33
4 Mark Haines <mark at matrix.org>
5 * HS core
6 * Crypto
7 * Content repository
8 * CS v2 API impl
9
10 Kegan Dougal <kegan at matrix.org>
11 * HS core
12 * CS v1 API impl
13 * AS API impl
14
15 Paul "LeoNerd" Evans <paul at matrix.org>
16 * HS core
17 * Presence
18 * Typing Notifications
19 * Performance metrics and caching layer
20
21 Dave Baker <dave at matrix.org>
22 * Push notifications
23 * Auth CS v2 impl
24
25 Matthew Hodgson <matthew at matrix.org>
26 * General doc & housekeeping
27 * Vertobot/vertobridge matrix<->verto PoC
28
29 Emmanuel Rohee <manu at matrix.org>
30 * Supporting iOS clients (testability and fallback registration)
4 ----
315
326 Turned to Dust <dwinslow86 at gmail.com>
337 * ArchLinux installation instructions
6135 * Add LDAP support for authentication
6236
6337 Pierre Jaury <pierre at jaury.eu>
64 * Docker packaging
38 * Docker packaging
6539
6640 Serban Constantin <serban.constantin at gmail dot com>
6741 * Small bug fix
6842
69 Jason Robinson <jasonr at matrix.org>
70 * Minor fixes
71
7243 Joseph Weston <joseph at weston.cloud>
73 + Add admin API for querying HS version
44 * Add admin API for querying HS version
7445
7546 Benjamin Saunders <ben.e.saunders at gmail dot com>
7647 * Documentation improvements
0 Synapse 1.5.0 (2019-10-29)
1 ==========================
2
3 Security updates
4 ----------------
5
6 This release includes a security fix ([\#6262](https://github.com/matrix-org/synapse/issues/6262), below). Administrators are encouraged to upgrade as soon as possible.
7
8 Bugfixes
9 --------
10
11 - Fix bug where room directory search was case sensitive. ([\#6268](https://github.com/matrix-org/synapse/issues/6268))
12
13
14 Synapse 1.5.0rc2 (2019-10-28)
15 =============================
16
17 Bugfixes
18 --------
19
20 - Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
21 - Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
22 - Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))
23
24
25 Internal Changes
26 ----------------
27
28 - Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
29 - Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
30
31
032 Synapse 1.5.0rc1 (2019-10-24)
133 ==========================
2
3 This release includes a database migration step **which may take a long time to complete**:
4
5 - Allow devices to be marked as hidden, for use by features such as cross-signing.
6 This adds a new field with a default value to the devices field in the database,
7 and so the database upgrade may take a long time depending on how many devices
8 are in the database. ([\#5759](https://github.com/matrix-org/synapse/issues/5759))
934
1035 Features
1136 --------
5075 ----------------
5176
5277 - Update `user_filters` table to have a unique index, and non-null columns. Thanks to @pik for contributing this. ([\#1172](https://github.com/matrix-org/synapse/issues/1172), [\#6175](https://github.com/matrix-org/synapse/issues/6175), [\#6184](https://github.com/matrix-org/synapse/issues/6184))
78 - Allow devices to be marked as hidden, for use by features such as cross-signing.
79 This adds a new field with a default value to the devices field in the database,
80 and so the database upgrade may take a long time depending on how many devices
81 are in the database. ([\#5759](https://github.com/matrix-org/synapse/issues/5759))
5382 - Move lookup-related functions from RoomMemberHandler to IdentityHandler. ([\#5978](https://github.com/matrix-org/synapse/issues/5978))
5483 - Improve performance of the public room list directory. ([\#6019](https://github.com/matrix-org/synapse/issues/6019), [\#6152](https://github.com/matrix-org/synapse/issues/6152), [\#6153](https://github.com/matrix-org/synapse/issues/6153), [\#6154](https://github.com/matrix-org/synapse/issues/6154))
5584 - Edit header dicts docstrings in `SimpleHttpClient` to note that `str` or `bytes` can be passed as header keys. ([\#6077](https://github.com/matrix-org/synapse/issues/6077))
113113 debian changelog. (Though typically such changes should be submitted as two
114114 separate pull requests.)
115115
116 Attribution
117 ~~~~~~~~~~~
118
119 Everyone who contributes anything to Matrix is welcome to be listed in the
120 AUTHORS.rst file for the project in question. Please feel free to include a
121 change to AUTHORS.rst in your pull request to list yourself and a short
122 description of the area(s) you've worked on. Also, we sometimes have swag to
123 give away to contributors - if you feel that Matrix-branded apparel is missing
124 from your life, please mail us your shipping address to matrix at matrix.org and
125 we'll try to fix it :)
126
127116 Sign off
128117 ~~~~~~~~
129118
11 =================
22
33 Before upgrading check if any special steps are required to upgrade from the
4 what you currently have installed to current version of Synapse. The extra
4 version you currently have installed to the current version of Synapse. The extra
55 instructions that may be required are listed later in this document.
66
77 * If Synapse was installed using `prebuilt packages
2828 running:
2929
3030 .. code:: bash
31
31
3232 git pull
3333 pip install --upgrade .
3434
7373 # replace `1.3.0` and `stretch` accordingly:
7474 wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
7575 dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
76
77
78 Upgrading to v1.5.0
79 ===================
80
81 This release includes a database migration which may take several minutes to
82 complete if there are a large number (more than a million or so) of entries in
83 the ``devices`` table. This is only likely to a be a problem on very large
84 installations.
85
7686
7787 Upgrading to v1.4.0
7888 ===================
0 matrix-synapse-py3 (1.5.0) stable; urgency=medium
1
2 * New synapse release 1.5.0.
3
4 -- Synapse Packaging team <packages@matrix.org> Tue, 29 Oct 2019 14:28:41 +0000
5
06 matrix-synapse-py3 (1.4.1) stable; urgency=medium
17
28 * New synapse release 1.4.1.
6767 "presence_list": ["accepted"],
6868 "presence_stream": ["currently_active"],
6969 "public_room_list_stream": ["visibility"],
70 "devices": ["hidden"],
7071 "device_lists_outbound_pokes": ["sent"],
7172 "users_who_share_rooms": ["share_private"],
7273 "groups": ["is_public"],
3535 except ImportError:
3636 pass
3737
38 __version__ = "1.5.0rc1"
38 __version__ = "1.5.0"
3939
4040 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
4141 # We import here so that we don't have to install a bunch of deps when
277277 pdu_to_check.sender_domain,
278278 e.getErrorMessage(),
279279 )
280 # XX not really sure if these are the right codes, but they are what
281 # we've done for ages
282 raise SynapseError(400, errmsg, Codes.UNAUTHORIZED)
280 raise SynapseError(403, errmsg, Codes.FORBIDDEN)
283281
284282 for p, d in zip(pdus_to_check_sender, more_deferreds):
285283 d.addErrback(sender_err, p)
313311 "event id %s: unable to verify signature for event id domain: %s"
314312 % (pdu_to_check.pdu.event_id, e.getErrorMessage())
315313 )
316 # XX as above: not really sure if these are the right codes
317 raise SynapseError(400, errmsg, Codes.UNAUTHORIZED)
314 raise SynapseError(403, errmsg, Codes.FORBIDDEN)
318315
319316 for p, d in zip(pdus_to_check_event_id, more_deferreds):
320317 d.addErrback(event_err, p)
369369 pdu = event_from_pdu_json(content, format_ver)
370370 origin_host, _ = parse_server_name(origin)
371371 yield self.check_server_matches_acl(origin_host, pdu.room_id)
372 pdu = yield self._check_sigs_and_hash(room_version, pdu)
372373 ret_pdu = yield self.handler.on_invite_request(origin, pdu)
373374 time_now = self._clock.time_msec()
374375 return {"event": ret_pdu.get_pdu_json(time_now)}
385386 yield self.check_server_matches_acl(origin_host, pdu.room_id)
386387
387388 logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
389
390 pdu = yield self._check_sigs_and_hash(room_version, pdu)
391
388392 res_pdus = yield self.handler.on_send_join_request(origin, pdu)
389393 time_now = self._clock.time_msec()
390394 return (
420424 yield self.check_server_matches_acl(origin_host, pdu.room_id)
421425
422426 logger.debug("on_send_leave_request: pdu sigs: %s", pdu.signatures)
427
428 pdu = yield self._check_sigs_and_hash(room_version, pdu)
429
423430 yield self.handler.on_send_leave_request(origin, pdu)
424431 return 200, {}
425432
12211221 Returns:
12221222 Deferred[FrozenEvent]
12231223 """
1224
12251224 if get_domain_from_id(user_id) != origin:
12261225 logger.info(
12271226 "Got /make_join request for user %r from different origin %s, ignoring",
12791278 event = pdu
12801279
12811280 logger.debug(
1282 "on_send_join_request: Got event: %s, signatures: %s",
1281 "on_send_join_request from %s: Got event: %s, signatures: %s",
1282 origin,
12831283 event.event_id,
12841284 event.signatures,
12851285 )
1286
1287 if get_domain_from_id(event.sender) != origin:
1288 logger.info(
1289 "Got /send_join request for user %r from different origin %s",
1290 event.sender,
1291 origin,
1292 )
1293 raise SynapseError(403, "User not from origin", Codes.FORBIDDEN)
12861294
12871295 event.internal_metadata.outlier = False
12881296 # Send this event on behalf of the origin server.
15011509 event.event_id,
15021510 event.signatures,
15031511 )
1512
1513 if get_domain_from_id(event.sender) != origin:
1514 logger.info(
1515 "Got /send_leave request for user %r from different origin %s",
1516 event.sender,
1517 origin,
1518 )
1519 raise SynapseError(403, "User not from origin", Codes.FORBIDDEN)
15041520
15051521 event.internal_metadata.outlier = False
15061522
4444 self.is_mine_id = hs.is_mine_id
4545 self.stats_bucket_size = hs.config.stats_bucket_size
4646
47 self.stats_enabled = hs.config.stats_enabled
48
4749 # The current position in the current_state_delta stream
4850 self.pos = None
4951
6062 def notify_new_event(self):
6163 """Called when there may be more deltas to process
6264 """
63 if not self.hs.config.stats_enabled or self._is_processing:
65 if not self.stats_enabled or self._is_processing:
6466 return
6567
6668 self._is_processing = True
246246 return result
247247
248248 return self.runInteraction("count_e2e_one_time_keys", _count_e2e_one_time_keys)
249
250 def _get_e2e_cross_signing_key_txn(self, txn, user_id, key_type, from_user_id=None):
251 """Returns a user's cross-signing key.
252
253 Args:
254 txn (twisted.enterprise.adbapi.Connection): db connection
255 user_id (str): the user whose key is being requested
256 key_type (str): the type of key that is being set: either 'master'
257 for a master key, 'self_signing' for a self-signing key, or
258 'user_signing' for a user-signing key
259 from_user_id (str): if specified, signatures made by this user on
260 the key will be included in the result
261
262 Returns:
263 dict of the key data or None if not found
264 """
265 sql = (
266 "SELECT keydata "
267 " FROM e2e_cross_signing_keys "
268 " WHERE user_id = ? AND keytype = ? ORDER BY stream_id DESC LIMIT 1"
269 )
270 txn.execute(sql, (user_id, key_type))
271 row = txn.fetchone()
272 if not row:
273 return None
274 key = json.loads(row[0])
275
276 device_id = None
277 for k in key["keys"].values():
278 device_id = k
279
280 if from_user_id is not None:
281 sql = (
282 "SELECT key_id, signature "
283 " FROM e2e_cross_signing_signatures "
284 " WHERE user_id = ? "
285 " AND target_user_id = ? "
286 " AND target_device_id = ? "
287 )
288 txn.execute(sql, (from_user_id, user_id, device_id))
289 row = txn.fetchone()
290 if row:
291 key.setdefault("signatures", {}).setdefault(from_user_id, {})[
292 row[0]
293 ] = row[1]
294
295 return key
296
297 def get_e2e_cross_signing_key(self, user_id, key_type, from_user_id=None):
298 """Returns a user's cross-signing key.
299
300 Args:
301 user_id (str): the user whose self-signing key is being requested
302 key_type (str): the type of cross-signing key to get
303 from_user_id (str): if specified, signatures made by this user on
304 the self-signing key will be included in the result
305
306 Returns:
307 dict of the key data or None if not found
308 """
309 return self.runInteraction(
310 "get_e2e_cross_signing_key",
311 self._get_e2e_cross_signing_key_txn,
312 user_id,
313 key_type,
314 from_user_id,
315 )
249316
250317
251318 class EndToEndKeyStore(EndToEndKeyWorkerStore, SQLBaseStore):
425492 key,
426493 )
427494
428 def _get_e2e_cross_signing_key_txn(self, txn, user_id, key_type, from_user_id=None):
429 """Returns a user's cross-signing key.
430
431 Args:
432 txn (twisted.enterprise.adbapi.Connection): db connection
433 user_id (str): the user whose key is being requested
434 key_type (str): the type of key that is being set: either 'master'
435 for a master key, 'self_signing' for a self-signing key, or
436 'user_signing' for a user-signing key
437 from_user_id (str): if specified, signatures made by this user on
438 the key will be included in the result
439
440 Returns:
441 dict of the key data or None if not found
442 """
443 sql = (
444 "SELECT keydata "
445 " FROM e2e_cross_signing_keys "
446 " WHERE user_id = ? AND keytype = ? ORDER BY stream_id DESC LIMIT 1"
447 )
448 txn.execute(sql, (user_id, key_type))
449 row = txn.fetchone()
450 if not row:
451 return None
452 key = json.loads(row[0])
453
454 device_id = None
455 for k in key["keys"].values():
456 device_id = k
457
458 if from_user_id is not None:
459 sql = (
460 "SELECT key_id, signature "
461 " FROM e2e_cross_signing_signatures "
462 " WHERE user_id = ? "
463 " AND target_user_id = ? "
464 " AND target_device_id = ? "
465 )
466 txn.execute(sql, (from_user_id, user_id, device_id))
467 row = txn.fetchone()
468 if row:
469 key.setdefault("signatures", {}).setdefault(from_user_id, {})[
470 row[0]
471 ] = row[1]
472
473 return key
474
475 def get_e2e_cross_signing_key(self, user_id, key_type, from_user_id=None):
476 """Returns a user's cross-signing key.
477
478 Args:
479 user_id (str): the user whose self-signing key is being requested
480 key_type (str): the type of cross-signing key to get
481 from_user_id (str): if specified, signatures made by this user on
482 the self-signing key will be included in the result
483
484 Returns:
485 dict of the key data or None if not found
486 """
487 return self.runInteraction(
488 "get_e2e_cross_signing_key",
489 self._get_e2e_cross_signing_key_txn,
490 user_id,
491 key_type,
492 from_user_id,
493 )
494
495495 def store_e2e_cross_signing_signatures(self, user_id, signatures):
496496 """Stores cross-signing signatures.
497497
200200 where_clauses.append(
201201 """
202202 (
203 name LIKE ?
204 OR topic LIKE ?
205 OR canonical_alias LIKE ?
203 LOWER(name) LIKE ?
204 OR LOWER(topic) LIKE ?
205 OR LOWER(canonical_alias) LIKE ?
206206 )
207207 """
208208 )
209 query_args += [search_term, search_term, search_term]
209 query_args += [
210 search_term.lower(),
211 search_term.lower(),
212 search_term.lower(),
213 ]
210214
211215 where_clause = ""
212216 if where_clauses:
0 /* Copyright 2019 New Vector Ltd
1 *
2 * Licensed under the Apache License, Version 2.0 (the "License");
3 * you may not use this file except in compliance with the License.
4 * You may obtain a copy of the License at
5 *
6 * http://www.apache.org/licenses/LICENSE-2.0
7 *
8 * Unless required by applicable law or agreed to in writing, software
9 * distributed under the License is distributed on an "AS IS" BASIS,
10 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 * See the License for the specific language governing permissions and
12 * limitations under the License.
13 */
14
15 -- device list needs to know which ones are "real" devices, and which ones are
16 -- just used to avoid collisions
17 ALTER TABLE devices ADD COLUMN hidden BOOLEAN DEFAULT FALSE;
0 /* Copyright 2019 New Vector Ltd
1 *
2 * Licensed under the Apache License, Version 2.0 (the "License");
3 * you may not use this file except in compliance with the License.
4 * You may obtain a copy of the License at
5 *
6 * http://www.apache.org/licenses/LICENSE-2.0
7 *
8 * Unless required by applicable law or agreed to in writing, software
9 * distributed under the License is distributed on an "AS IS" BASIS,
10 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 * See the License for the specific language governing permissions and
12 * limitations under the License.
13 */
14
15 -- cross-signing keys
16 CREATE TABLE IF NOT EXISTS e2e_cross_signing_keys (
17 user_id TEXT NOT NULL,
18 -- the type of cross-signing key (master, user_signing, or self_signing)
19 keytype TEXT NOT NULL,
20 -- the full key information, as a json-encoded dict
21 keydata TEXT NOT NULL,
22 -- for keeping the keys in order, so that we can fetch the latest one
23 stream_id BIGINT NOT NULL
24 );
25
26 CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys(user_id, keytype, stream_id);
27
28 -- cross-signing signatures
29 CREATE TABLE IF NOT EXISTS e2e_cross_signing_signatures (
30 -- user who did the signing
31 user_id TEXT NOT NULL,
32 -- key used to sign
33 key_id TEXT NOT NULL,
34 -- user who was signed
35 target_user_id TEXT NOT NULL,
36 -- device/key that was signed
37 target_device_id TEXT NOT NULL,
38 -- the actual signature
39 signature TEXT NOT NULL
40 );
41
42 CREATE UNIQUE INDEX e2e_cross_signing_signatures_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);
43
44 -- stream of user signature updates
45 CREATE TABLE IF NOT EXISTS user_signature_stream (
46 -- uses the same stream ID as device list stream
47 stream_id BIGINT NOT NULL,
48 -- user who did the signing
49 from_user_id TEXT NOT NULL,
50 -- list of users who were signed, as a JSON array
51 user_ids TEXT NOT NULL
52 );
53
54 CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream(stream_id);
+0
-18
synapse/storage/schema/delta/56/hidden_devices.sql less more
0 /* Copyright 2019 New Vector Ltd
1 *
2 * Licensed under the Apache License, Version 2.0 (the "License");
3 * you may not use this file except in compliance with the License.
4 * You may obtain a copy of the License at
5 *
6 * http://www.apache.org/licenses/LICENSE-2.0
7 *
8 * Unless required by applicable law or agreed to in writing, software
9 * distributed under the License is distributed on an "AS IS" BASIS,
10 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 * See the License for the specific language governing permissions and
12 * limitations under the License.
13 */
14
15 -- device list needs to know which ones are "real" devices, and which ones are
16 -- just used to avoid collisions
17 ALTER TABLE devices ADD COLUMN hidden BOOLEAN DEFAULT FALSE;
+0
-55
synapse/storage/schema/delta/56/signing_keys.sql less more
0 /* Copyright 2019 New Vector Ltd
1 *
2 * Licensed under the Apache License, Version 2.0 (the "License");
3 * you may not use this file except in compliance with the License.
4 * You may obtain a copy of the License at
5 *
6 * http://www.apache.org/licenses/LICENSE-2.0
7 *
8 * Unless required by applicable law or agreed to in writing, software
9 * distributed under the License is distributed on an "AS IS" BASIS,
10 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 * See the License for the specific language governing permissions and
12 * limitations under the License.
13 */
14
15 -- cross-signing keys
16 CREATE TABLE IF NOT EXISTS e2e_cross_signing_keys (
17 user_id TEXT NOT NULL,
18 -- the type of cross-signing key (master, user_signing, or self_signing)
19 keytype TEXT NOT NULL,
20 -- the full key information, as a json-encoded dict
21 keydata TEXT NOT NULL,
22 -- for keeping the keys in order, so that we can fetch the latest one
23 stream_id BIGINT NOT NULL
24 );
25
26 CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys(user_id, keytype, stream_id);
27
28 -- cross-signing signatures
29 CREATE TABLE IF NOT EXISTS e2e_cross_signing_signatures (
30 -- user who did the signing
31 user_id TEXT NOT NULL,
32 -- key used to sign
33 key_id TEXT NOT NULL,
34 -- user who was signed
35 target_user_id TEXT NOT NULL,
36 -- device/key that was signed
37 target_device_id TEXT NOT NULL,
38 -- the actual signature
39 signature TEXT NOT NULL
40 );
41
42 CREATE UNIQUE INDEX e2e_cross_signing_signatures_idx ON e2e_cross_signing_signatures(user_id, target_user_id, target_device_id);
43
44 -- stream of user signature updates
45 CREATE TABLE IF NOT EXISTS user_signature_stream (
46 -- uses the same stream ID as device list stream
47 stream_id BIGINT NOT NULL,
48 -- user who did the signing
49 from_user_id TEXT NOT NULL,
50 -- list of users who were signed, as a JSON array
51 user_ids TEXT NOT NULL
52 );
53
54 CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream(stream_id);
606606 """
607607
608608 self.hs.config.stats_enabled = False
609 self.handler.stats_enabled = False
609610
610611 u1 = self.register_user("u1", "pass")
611612 u1token = self.login("u1", "pass")
617618 self.assertIsNone(self._get_current_stats("user", u1))
618619
619620 self.hs.config.stats_enabled = True
621 self.handler.stats_enabled = True
620622
621623 self._perform_background_initial_update()
622624
113113 basepython = python3.6
114114 deps =
115115 flake8
116 black
116 black==19.3b0 # We pin so that our tests don't start failing on new releases of black.
117117 commands =
118118 python -m black --check --diff .
119119 /bin/sh -c "flake8 synapse tests scripts scripts-dev scripts/hash_password scripts/register_new_matrix_user scripts/synapse_port_db synctl {env:PEP8SUFFIX:}"