Commit dba2a81f87ac6ee5baf58f0654e4ef154aed7b4d - policyd-rate-limit

+1

-1

Makefile less more

8	8	[ ! -f /etc/policyd-rate-limit.yaml ] && cp -n policyd_rate_limit/policyd-rate-limit.yaml /etc/ \|\| true
9	9	cp -n init/policyd-rate-limit /etc/init.d
10	10	cp -n init/policyd-rate-limit.service /etc/systemd/system/ \|\| true
11		pip3 install policyd-rate-limit -U --force-reinstall --no-deps --no-binary :all -f ./dist/policyd-rate-limit-${VERSION}.tar.gz
	11	pip3 install policyd-rate-limit --no-cache-dir -U --force-reinstall --no-deps --no-binary :all -f ./dist/policyd-rate-limit-${VERSION}.tar.gz
12	12	systemctl daemon-reload
13	13	uninstall:
14	14	pip3 uninstall policyd-rate-limit \|\| true

+28

-7

PKG-INFO less more

0	0	Metadata-Version: 1.1
1	1	Name: policyd-rate-limit
2		Version: 0.7.1
	2	Version: 1.0.0
3	3	Summary: Postfix rate limit policy server implemented in Python3.
4	4	Home-page: https://github.com/nitmir/policyd-rate-limit
5	5	Author: Valentin Samir

57	57
58	58	0 0 * * * policyd-rate-limit /usr/local/bin/policyd-rate-limit --clean >/dev/null
59	59
	60
	61	Options of the ``policyd-rate-limit`` binary
	62	--------------------------------------------
	63
	64	* ``-h``, ``--help``: show the help message and exit
	65	* ``--clean``: clean old records from the database
	66	* ``--get-config PARAMETER_NAME`` return the value of a config parameter
	67	* ``--file CONFIG_PATH``, ``-f CONFIG_PATH``: path to a config file
	68
60	69	Settings
61	70	--------
62	71
63		``policyd-rate-limit`` search for its config first in ``~/.config/policyd-rate-limit.conf``
64		If not found, then in ``/etc/policyd-rate-limit.conf``, and if not found use the default config.
	72	If the option ``--file`` is not specified, ``policyd-rate-limit`` try to read its configuration from
	73	the following path and choose the first existing file:
	74
	75	* ~/.config/policyd-rate-limit.conf
	76	* ~/.config/policyd-rate-limit.yaml
	77	* /etc/policyd-rate-limit.conf
	78	* /etc/policyd-rate-limit.yaml
	79
	80	The ``.conf`` are the old configuration format. It was a python module and should not be used.
	81	The ``.yaml`` are the new configuration format using the YAML syntax.
	82
65	83
66	84	* ``debug``: make ``policyd-rate-limit`` output logs to stderr.
67	85	The default is ``True``.

85	103	* ``limits``: A list of couple [number of emails, number of seconds]. If one of the element of the
86	104	list is exeeded (more than 'number of emails' on 'number of seconds' for an ip address or an sasl
87	105	username), postfix will return a temporary failure.
88		* ``limits_by_id``: A dictionnary of id -> limit list (see limits). Used to override limits and use
	106	* ``limits_by_id``: A dictionary of id -> limit list (see limits). Used to override limits and use
89	107	custom limits for a particular id. Use an empty list for no limits for a particular id.
90	108	Ids are sasl usernames or ip addresses. The default is ``{}``.
91	109	* ``limit_by_sasl``: Apply limits by sasl usernames. The default is ``True``.
92		* ``limit_by_ip``: Apply limits by ip addresses if sasl username is not found.
	110	* ``limit_by_sender``: Apply limits by sender addresses if sasl username is not found.
	111	The defaut is ``False``.
	112	* ``limit_by_ip``: Apply limits by ip addresses if sasl username and sender address are not found.
93	113	The default is ``False``.
94	114	* ``limited_networks``: A list of ip networks in cidr notation on which limits are applied. An empty
95	115	list is equal to ``limit_by_ip = False``, put ``"0.0.0.0/0"`` and ``::/0`` for every ip addresses.

110	130	--clean is called. The default is ``False``.
111	131	* ``report_from``: From who to send emails reports. It must be defined when ``report`` is ``True``.
112	132	* ``report_to``: Address to send emails reports to. It must be defined when ``report`` is ``True``.
	133	It can be a single email address or a list of email adresses.
113	134	* ``report_subject``: Subject of the report email. The default is ``"policyd-rate-limit report"``.
114	135	* ``report_limits``: List of number of seconds from the limits list for which you want to be reported.
115	136	The default is ``[86400]``.

154	175	.. \|travis\| image:: https://badges.genua.fr/travis/nitmir/policyd-rate-limit/master.svg
155	176	:target: https://travis-ci.org/nitmir/policyd-rate-limit
156	177
157		.. \|coverage\| image:: https://badges.genua.fr/local/coverage/?project=policyd-rate-limit
158		:target: https://badges.genua.fr/local/coverage/policyd-rate-limit/
	178	.. \|coverage\| image:: https://badges.genua.fr/coverage/badge/policyd-rate-limit/master.svg
	179	:target: https://badges.genua.fr/coverage/policyd-rate-limit/
159	180
160	181	.. \|pypi_version\| image:: https://badges.genua.fr/pypi/v/policyd-rate-limit.svg
161	182	:target: https://pypi.python.org/pypi/policyd-rate-limit

+27

-6

README.rst less more

48	48
49	49	0 0 * * * policyd-rate-limit /usr/local/bin/policyd-rate-limit --clean >/dev/null
50	50
	51
	52	Options of the ``policyd-rate-limit`` binary
	53	--------------------------------------------
	54
	55	* ``-h``, ``--help``: show the help message and exit
	56	* ``--clean``: clean old records from the database
	57	* ``--get-config PARAMETER_NAME`` return the value of a config parameter
	58	* ``--file CONFIG_PATH``, ``-f CONFIG_PATH``: path to a config file
	59
51	60	Settings
52	61	--------
53	62
54		``policyd-rate-limit`` search for its config first in ``~/.config/policyd-rate-limit.conf``
55		If not found, then in ``/etc/policyd-rate-limit.conf``, and if not found use the default config.
	63	If the option ``--file`` is not specified, ``policyd-rate-limit`` try to read its configuration from
	64	the following path and choose the first existing file:
	65
	66	* ~/.config/policyd-rate-limit.conf
	67	* ~/.config/policyd-rate-limit.yaml
	68	* /etc/policyd-rate-limit.conf
	69	* /etc/policyd-rate-limit.yaml
	70
	71	The ``.conf`` are the old configuration format. It was a python module and should not be used.
	72	The ``.yaml`` are the new configuration format using the YAML syntax.
	73
56	74
57	75	* ``debug``: make ``policyd-rate-limit`` output logs to stderr.
58	76	The default is ``True``.

76	94	* ``limits``: A list of couple [number of emails, number of seconds]. If one of the element of the
77	95	list is exeeded (more than 'number of emails' on 'number of seconds' for an ip address or an sasl
78	96	username), postfix will return a temporary failure.
79		* ``limits_by_id``: A dictionnary of id -> limit list (see limits). Used to override limits and use
	97	* ``limits_by_id``: A dictionary of id -> limit list (see limits). Used to override limits and use
80	98	custom limits for a particular id. Use an empty list for no limits for a particular id.
81	99	Ids are sasl usernames or ip addresses. The default is ``{}``.
82	100	* ``limit_by_sasl``: Apply limits by sasl usernames. The default is ``True``.
83		* ``limit_by_ip``: Apply limits by ip addresses if sasl username is not found.
	101	* ``limit_by_sender``: Apply limits by sender addresses if sasl username is not found.
	102	The defaut is ``False``.
	103	* ``limit_by_ip``: Apply limits by ip addresses if sasl username and sender address are not found.
84	104	The default is ``False``.
85	105	* ``limited_networks``: A list of ip networks in cidr notation on which limits are applied. An empty
86	106	list is equal to ``limit_by_ip = False``, put ``"0.0.0.0/0"`` and ``::/0`` for every ip addresses.

101	121	--clean is called. The default is ``False``.
102	122	* ``report_from``: From who to send emails reports. It must be defined when ``report`` is ``True``.
103	123	* ``report_to``: Address to send emails reports to. It must be defined when ``report`` is ``True``.
	124	It can be a single email address or a list of email adresses.
104	125	* ``report_subject``: Subject of the report email. The default is ``"policyd-rate-limit report"``.
105	126	* ``report_limits``: List of number of seconds from the limits list for which you want to be reported.
106	127	The default is ``[86400]``.

145	166	.. \|travis\| image:: https://badges.genua.fr/travis/nitmir/policyd-rate-limit/master.svg
146	167	:target: https://travis-ci.org/nitmir/policyd-rate-limit
147	168
148		.. \|coverage\| image:: https://badges.genua.fr/local/coverage/?project=policyd-rate-limit
149		:target: https://badges.genua.fr/local/coverage/policyd-rate-limit/
	169	.. \|coverage\| image:: https://badges.genua.fr/coverage/badge/policyd-rate-limit/master.svg
	170	:target: https://badges.genua.fr/coverage/policyd-rate-limit/
150	171
151	172	.. \|pypi_version\| image:: https://badges.genua.fr/pypi/v/policyd-rate-limit.svg
152	173	:target: https://pypi.python.org/pypi/policyd-rate-limit

+4

-1

docs/policyd-rate-limit.yaml.5.rst less more

67	67	exceeded (more than 'number of emails' on 'number of seconds' for an ip address or an sasl
68	68	username), postfix will return a temporary failure.
69	69	limits_by_id
70		A dictionnary of id -> limit list (see limits). Used to override limits and use custom limits for
	70	A dictionary of id -> limit list (see limits). Used to override limits and use custom limits for
71	71	a particular id. Use an empty list for no limits for a particular id. Ids are sasl usernames or
72	72	ip addresses. The default is {}.
73	73	limit_by_sasl
74	74	Apply limits by sasl usernames. The default is True.
	75	limit_by_sender
	76	Apply limits by sender addresses if sasl username is not found. The defaut is ``False``.
75	77	limit_by_ip
76	78	Apply limits by ip addresses if sasl username is not found. The default is False.
77	79	limited_networks

100	102	From who to send emails reports. It must be defined when report is True.
101	103	report_to
102	104	Address to send emails reports to. It must be defined when report is True.
	105	It can be a single email address or a list of email adresses.
103	106	report_subject
104	107	Subject of the report email. The default is "policyd-rate-limit report".
105	108	report_limits

+5

-1

policyd_rate_limit/config.py less more

52	52	limits_by_id = {}
53	53
54	54	limit_by_sasl = True
	55	limit_by_sender = False
55	56	limit_by_ip = False
56	57
57	58	limited_networks = []

67	68	report = False
68	69	# from who to send emails reports
69	70	report_from = None
70		# address to send emails reports to
	71	# address to send emails reports to. It can be a single email or a list of emails
71	72	report_to = None
72	73	# subject of the report email
73	74	report_subject = "policyd-rate-limit report"

82	83	smtp_starttls = False
83	84	# Should we use credentials to connect to smtp_server ? if yes set ("user", "password"), else None
84	85	smtp_credentials = None
	86
	87	# The time in seconds before an unused socket gets closed
	88	delay_to_close = 300

+1

-1

policyd_rate_limit/policyd-rate-limit.conf less more

57	57	report = False
58	58	# from who to send emails reports
59	59	report_from = None
60		# address to send emails reports to
	60	# address to send emails reports to. It can be a single email or a list of emails
61	61	report_to = None
62	62	# subject of the report email
63	63	report_subject = "policyd-rate-limit report"

+13

-1

policyd_rate_limit/policyd-rate-limit.yaml less more

55	55
56	56	# Apply limits by sasl usernames.
57	57	limit_by_sasl: True
58		# If no sasl username is found, apply limits by ip addresses.
	58	# If no sasl username is found, or limit by sasl usernames disabled,
	59	# apply limits by sender addresses.
	60	limit_by_sender: False
	61	# If sasl username and sender address not found or disabled, apply limits by ip addresses.
59	62	limit_by_ip: False
60	63
61	64	# A list of ip networks in cidr notation on which limits are applied. An empty list is equal

77	80	# from who to send emails reports. Must be defined if report: True
78	81	report_from: null
79	82	# Address to send emails reports to. Must be defined if report: True
	83	# If can be a single email address:
	84	# report_to: foo@example.com
	85	# or a list of email addresses:
	86	# report_to:
	87	# - foo@example.com
	88	# - bar@example.com
80	89	report_to: null
81	90	# Subject of the report email
82	91	report_subject: "policyd-rate-limit report"

92	101	smtp_starttls: False
93	102	# Should we use credentials to connect to smtp_server ? if yes set ["user", "password"], else null
94	103	smtp_credentials: null
	104
	105	# The time in seconds before an unused socket gets closed
	106	delay_to_close: 300

+42

-3

policyd_rate_limit/policyd.py less more

12	12	import socket
13	13	import time
14	14	import select
	15	import traceback
15	16
16	17	from policyd_rate_limit import utils
17	18	from policyd_rate_limit.utils import config

25	26	"""The policy server class"""
26	27	socket_data_read = {}
27	28	socket_data_write = {}
	29	last_used = {}
28	30
29	31	def socket(self):
30	32	"""initialize the socket from the config parameters"""

69	71	except KeyError:
70	72	pass
71	73	connection.close()
	74
	75	def close_write_conn(self, connection):
	76	"""Removes a socket from the write dict"""
	77	try:
	78	del self.socket_data_write[connection]
	79	except KeyError:
	80	if config.debug:
	81	sys.stderr.write(
	82	(
	83	"Hmmm, a socket actually used to write a little "
	84	"time ago wasn\'t in socket_data_write. Weird.\n"
	85	)
	86	)
72	87
73	88	def run(self):
74	89	"""The main server loop"""

95	110	sys.stderr.write('connection from %s\n' % (client_address,))
96	111	sys.stderr.flush()
97	112	self.socket_data_read[connection] = []
	113
	114	# Updates the last_sed time for the socket.
	115	self.last_used[connection] = time.time()
98	116	# else there is data to read on a client socket
99	117	else:
100	118	self.read(socket)

106	124	if data_not_sent:
107	125	self.socket_data_write[socket] = data_not_sent
108	126	else:
109		self.close_connection(socket)
	127	self.close_write_conn(socket)
	128
	129	# Socket has been used, let's update its last_used time.
	130	self.last_used[socket] = time.time()
110	131	# the socket has been closed during read
111	132	except KeyError:
112	133	pass
	134	# Closes unused socket for a long time.
	135	__to_rm = []
	136	for (socket, last_used) in self.last_used.items():
	137	if socket == sock:
	138	continue
	139	if time.time() - last_used > config.delay_to_close:
	140	self.close_connection(socket)
	141	__to_rm.append(socket)
	142	for socket in __to_rm:
	143	self.last_used.pop(socket)
	144
113	145	except (KeyboardInterrupt, utils.Exit):
114	146	for socket in list(self.socket_data_read.keys()):
115	147	if socket != self.sock:

153	185	self.action(connection, request)
154	186	else:
155	187	self.socket_data_read[connection] = buffer
	188	# Socket has been used, let's update its last_used time.
	189	self.last_used[connection] = time.time()
156	190	except (KeyboardInterrupt, utils.Exit):
157	191	self.close_connection(connection)
158	192	raise
159	193	except Exception as error:
160		sys.stderr.write("%s\n" % error)
	194	traceback.print_exc()
161	195	sys.stderr.flush()
162	196	self.close_connection(connection)
163	197

179	213	# if user is authenticated, we filter by sasl username
180	214	if config.limit_by_sasl and u'sasl_username' in request:
181	215	id = request[u'sasl_username']
	216	# else, if activated, we filter by sender
	217	elif config.limit_by_sender and u'sender' in request:
	218	id = request[u'sender']
182	219	# else, if activated, we filter by ip source addresse
183	220	elif (
184	221	config.limit_by_ip and

190	227	# to the next section
191	228	else:
192	229	raise Pass()
193		# Here we are limiting agains sasl username or ip source addresses.
	230	# Here we are limiting against sasl username, sender or source ip addresses.
194	231	# for each limit periods, we count the number of mails already send.
195	232	# if the a limit is reach, we change action to fail (deny the mail).
196	233	for mail_nb, delta in config.limits_by_id.get(id, config.limits):

231	268	sys.stderr.flush()
232	269	# return the result to the client
233	270	self.socket_data_write[connection] = data.encode('UTF-8')
	271	# Socket has been used, let's update its last_used time.
	272	self.last_used[connection] = time.time()

+2

-0

policyd_rate_limit/tests/utils.py less more

144	144	p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
145	145	launch_instance.i += 1
146	146	return (p, cfg_path)
	147
	148
147	149	launch_instance.i = 0
148	150
149	151

+29

-14

policyd_rate_limit/utils.py less more

267	267	def del_db(cls):
268	268	try:
269	269	cls._db[threading.current_thread()].close()
270		except:
	270	except Exception as e:
271	271	pass
272	272	try:
273	273	del cls._db[threading.current_thread()]

331	331	print("%d records deleted" % cur.rowcount)
332	332	# if report is True, generate a mail report
333	333	if config.report and config.report_to:
334		send_report(cur)
335		# The mail report has been successfully send, flush limit_report
336		cur.execute("DELETE FROM limit_report")
	334	report_text = gen_report(cur)
	335	# The mail report has been successfully send, flush limit_report
	336	cur.execute("DELETE FROM limit_report")
	337	# send report
	338	if len(report_text) != 0:
	339	send_report(report_text)
337	340
338	341	try:
339	342	if config.backend == PGSQL_DB:

354	357	cursor.get_db().autocommit = False
355	358
356	359
357		def send_report(cur):
	360	def gen_report(cur):
358	361	cur.execute("SELECT id, delta, hit FROM limit_report")
359	362	# list to sort ids by hits
360	363	report = list(cur.fetchall())
	364	text = []
361	365	if not config.report_only_if_needed or report:
362	366	if report:
363	367	text = ["Below is the table of users who hit a limit since the last cleanup:", ""]

367	371	for (id, delta, hit) in report:
368	372	report_d[id].append((delta, hit))
369	373	max_d['id'] = max(max_d['id'], len(id))
370		max_d['delta'] = max(max_d['delta'], len(str(delta)))
	374	max_d['delta'] = max(max_d['delta'], len(str(delta)) + 1)
371	375	max_d['hit'] = max(max_d['hit'], len(str(hit)))
372	376	# sort by hits
373	377	report.sort(key=lambda x: x[2])

403	407	else:
404	408	text = ["No user hit a limit since the last cleanup"]
405	409	text.extend(["", "-- ", "policyd-rate-limit"])
406
407		# Start building the mail report
408		msg = MIMEMultipart()
409		msg['Subject'] = config.report_subject or ""
410		msg['From'] = config.report_from or ""
411		msg['To'] = config.report_to
412		msg.attach(MIMEText("\n".join(text), 'plain'))
	410	return text
	411
	412
	413	def send_report(text):
413	414
414	415	# check that smtp_server is wekk formated
415	416	if isinstance(config.smtp_server, (list, tuple)):

435	436	server.login(config.smtp_credentials[0], config.smtp_credentials[1])
436	437	else:
437	438	ValueError("bad smtp_credentials should be a tuple (login, password)")
438		server.sendmail(config.report_from or "", config.report_to, msg.as_string())
	439
	440	if not isinstance(config.report_to, list):
	441	report_to = [config.report_to]
	442	else:
	443	report_to = config.report_to
	444	for rcpt in report_to:
	445	# Start building the mail report
	446	msg = MIMEMultipart()
	447	msg['Subject'] = config.report_subject or ""
	448	msg['From'] = config.report_from or ""
	449	msg['To'] = rcpt
	450	msg.attach(MIMEText("\n".join(text), 'plain'))
	451	server.sendmail(config.report_from or "", rcpt, msg.as_string())
439	452	finally:
	453	print('report is sent')
440	454	server.quit()
441	455
442	456

537	551	Used for coverage computation"""
538	552	raise Exit()
539	553
	554
540	555	config = LazyConfig()

+28

-7

policyd_rate_limit.egg-info/PKG-INFO less more

0	0	Metadata-Version: 1.1
1	1	Name: policyd-rate-limit
2		Version: 0.7.1
	2	Version: 1.0.0
3	3	Summary: Postfix rate limit policy server implemented in Python3.
4	4	Home-page: https://github.com/nitmir/policyd-rate-limit
5	5	Author: Valentin Samir

57	57
58	58	0 0 * * * policyd-rate-limit /usr/local/bin/policyd-rate-limit --clean >/dev/null
59	59
	60
	61	Options of the ``policyd-rate-limit`` binary
	62	--------------------------------------------
	63
	64	* ``-h``, ``--help``: show the help message and exit
	65	* ``--clean``: clean old records from the database
	66	* ``--get-config PARAMETER_NAME`` return the value of a config parameter
	67	* ``--file CONFIG_PATH``, ``-f CONFIG_PATH``: path to a config file
	68
60	69	Settings
61	70	--------
62	71
63		``policyd-rate-limit`` search for its config first in ``~/.config/policyd-rate-limit.conf``
64		If not found, then in ``/etc/policyd-rate-limit.conf``, and if not found use the default config.
	72	If the option ``--file`` is not specified, ``policyd-rate-limit`` try to read its configuration from
	73	the following path and choose the first existing file:
	74
	75	* ~/.config/policyd-rate-limit.conf
	76	* ~/.config/policyd-rate-limit.yaml
	77	* /etc/policyd-rate-limit.conf
	78	* /etc/policyd-rate-limit.yaml
	79
	80	The ``.conf`` are the old configuration format. It was a python module and should not be used.
	81	The ``.yaml`` are the new configuration format using the YAML syntax.
	82
65	83
66	84	* ``debug``: make ``policyd-rate-limit`` output logs to stderr.
67	85	The default is ``True``.

85	103	* ``limits``: A list of couple [number of emails, number of seconds]. If one of the element of the
86	104	list is exeeded (more than 'number of emails' on 'number of seconds' for an ip address or an sasl
87	105	username), postfix will return a temporary failure.
88		* ``limits_by_id``: A dictionnary of id -> limit list (see limits). Used to override limits and use
	106	* ``limits_by_id``: A dictionary of id -> limit list (see limits). Used to override limits and use
89	107	custom limits for a particular id. Use an empty list for no limits for a particular id.
90	108	Ids are sasl usernames or ip addresses. The default is ``{}``.
91	109	* ``limit_by_sasl``: Apply limits by sasl usernames. The default is ``True``.
92		* ``limit_by_ip``: Apply limits by ip addresses if sasl username is not found.
	110	* ``limit_by_sender``: Apply limits by sender addresses if sasl username is not found.
	111	The defaut is ``False``.
	112	* ``limit_by_ip``: Apply limits by ip addresses if sasl username and sender address are not found.
93	113	The default is ``False``.
94	114	* ``limited_networks``: A list of ip networks in cidr notation on which limits are applied. An empty
95	115	list is equal to ``limit_by_ip = False``, put ``"0.0.0.0/0"`` and ``::/0`` for every ip addresses.

110	130	--clean is called. The default is ``False``.
111	131	* ``report_from``: From who to send emails reports. It must be defined when ``report`` is ``True``.
112	132	* ``report_to``: Address to send emails reports to. It must be defined when ``report`` is ``True``.
	133	It can be a single email address or a list of email adresses.
113	134	* ``report_subject``: Subject of the report email. The default is ``"policyd-rate-limit report"``.
114	135	* ``report_limits``: List of number of seconds from the limits list for which you want to be reported.
115	136	The default is ``[86400]``.

154	175	.. \|travis\| image:: https://badges.genua.fr/travis/nitmir/policyd-rate-limit/master.svg
155	176	:target: https://travis-ci.org/nitmir/policyd-rate-limit
156	177
157		.. \|coverage\| image:: https://badges.genua.fr/local/coverage/?project=policyd-rate-limit
158		:target: https://badges.genua.fr/local/coverage/policyd-rate-limit/
	178	.. \|coverage\| image:: https://badges.genua.fr/coverage/badge/policyd-rate-limit/master.svg
	179	:target: https://badges.genua.fr/coverage/policyd-rate-limit/
159	180
160	181	.. \|pypi_version\| image:: https://badges.genua.fr/pypi/v/policyd-rate-limit.svg
161	182	:target: https://pypi.python.org/pypi/policyd-rate-limit

+1

-1

setup.py less more

36	36
37	37	setup(
38	38	name='policyd-rate-limit',
39		version='0.7.1',
	39	version='1.0.0',
40	40	description=DESC,
41	41	long_description=README,
42	42	author='Valentin Samir',