Commit a9f2422a6d8fd39653dbd0878918b66c3ecf8270 - balboa

Update upstream source from tag 'upstream/2.0.0+ds' Update to upstream version '2.0.0+ds' with Debian dir 6af36b575489e2ac6afd118ed6b40c6d199202e0 Sascha Steinbiss 4 years ago

62 changed file(s) with 20420 addition(s) and 2445 deletion(s). Raw diff Collapse all Expand all

+141

-104

README.md less more

0		# balboa [![CircleCI](https://circleci.com/gh/DCSO/balboa.svg?style=svg)](https://circleci.com/gh/DCSO/balboa)
1
2		balboa is the BAsic Little Book Of Answers. It consumes and indexes observations from [passive DNS](https://www.farsightsecurity.com/technical/passive-dns/) collection, providing a [GraphQL](https://graphql.org/) interface to access the aggregated contents of the observations database. We built balboa to handle passive DNS data aggregated from metadata gathered by [Suricata](https://suricata-ids.org).
3
4		The API should be suitable for integration into existing multi-source observable integration frameworks. It is possible to produce results in a [Common Output Format](https://datatracker.ietf.org/doc/draft-dulaunoy-dnsop-passive-dns-cof/) compatible schema using the GraphQL API. In fact, the GraphQL schema is modelled after the COF fields.
	0	# 📑 balboa [![CircleCI](https://circleci.com/gh/DCSO/balboa.svg?style=svg)](https://circleci.com/gh/DCSO/balboa)
	1
	2	balboa is the BAsic Little Book Of Answers. It consumes and indexes
	3	observations from [passive DNS](https://www.farsightsecurity.com/technical/passive-dns/)
	4	collection, providing a [GraphQL](https://graphql.org/) interface to access
	5	the aggregated contents of the observations database. We built balboa to handle
	6	passive DNS data aggregated from metadata gathered by
	7	[Suricata](https://suricata-ids.org).
	8
	9	The API should be suitable for integration into existing multi-source
	10	observable integration frameworks. It is possible to produce results in a
	11	[Common Output Format](https://datatracker.ietf.org/doc/draft-dulaunoy-dnsop-passive-dns-cof/)
	12	compatible schema using the GraphQL API. In fact, the GraphQL schema is
	13	modelled after the COF fields.
5	14
6	15	The balboa software...
7	16
8	17	- is fast for queries and input/updates
9	18	- implements persistent, compressed storage of observations
10		- as local storage
11		- in a [Cassandra](https://cassandra.apache.org) cluster (experimental)
12	19	- supports tracking and specifically querying multiple sensors
13	20	- makes use of multi-core systems
14	21	- can accept input from multiple sources simultaneously

18	25	- Unix socket
19	26	- accepts various (text-based) input formats
20	27	- JSON-based
21		- [FEVER](https://github.com/DCSO/fever)
22		- [gopassivedns](https://github.com/Phillipmartin/gopassivedns)
23		- [Packetbeat](https://www.elastic.co/guide/en/beats/packetbeat/master/packetbeat-dns-options.html) (via Logstash)
24		- [Suricata EVE DNS v1 and v2](http://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html#event-type-dns)
	28	- [FEVER](https://github.com/DCSO/fever)
	29	- [gopassivedns](https://github.com/Phillipmartin/gopassivedns)
	30	- [Packetbeat](https://www.elastic.co/guide/en/beats/packetbeat/master/packetbeat-dns-options.html) (via
	31	Logstash)
	32	- [Suricata EVE DNS v1 and v2](http://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html#event-type-dns)
25	33	- flat file
26		- Edward Fjellskål's [PassiveDNS](https://github.com/gamelinux/passivedns) tabular format (default order `-f SMcsCQTAtn`)
	34	- Edward Fjellskål's [PassiveDNS](https://github.com/gamelinux/passivedns) tabular format (default order `-f SMcsCQTAtn`)
27	35
28	36	## Building and Installation
29	37
30		```
	38	```text
31	39	$ go get github.com/DCSO/balboa
32		```
33
34		Dependencies:
	40	...
	41	```
	42
	43	To build the backends:
	44
	45	```text
	46	$ cd $GOPATH/src/github.com/DCSO/balboa/backends
	47	$ make
	48	...
	49	```
	50
	51	This will create a binary executable in the `build/` subdirectories of each backends directory.
	52
	53	### Dependencies
35	54
36	55	- Go 1.7 or later
37	56	- [RocksDB](https://rocksdb.org/) 5.0 or later (shared lib, with LZ4 support)
38		- [tpl](https://troydhanson.github.io/tpl/index.html) (shared lib)
39	57
40	58	On Debian (testing and stretch-backports), one can satisfy these dependencies with:
41	59
42		```
43		apt install librocksdb-dev libtpl-dev
	60	```text
	61	% apt install golang-go librocksdb-dev
	62	...
44	63	```
45	64
46	65	## Usage
47	66
48	67	### Configuring feeders
49	68
50		Feeders are used to get observations into the database. They run concurrently and process inputs in the background, making results accessible via the query interface as soon as the resulting upsert transactions have been completed in the database. What feeders are to be created is defined in a YAML configuration file (to be passed via the `-f` parameter to `balboa serve`). Example:
	69	Feeders are used to get observations into the database. They run concurrently
	70	and process inputs in the background, making results accessible via the query
	71	interface as soon as the resulting upsert transactions have been completed in
	72	the database. What feeders are to be created is defined in a YAML configuration
	73	file (to be passed via the `-f` parameter to `balboa serve`). Example:
51	74
52	75	```yaml
53	76	feeder:

67	90	input_format: gopassivedns
68	91	```
69	92
70		A balboa instance given this feeder configuration would support the following input options:
71
72		- JSON in FEVER's aggregate format delivered via AMQP from a temporary queue attached to the exchange `tdh.pdns` on `localhost` port 5762, authenticated with user `guest` and password `guest`
	93	A balboa instance given this feeder configuration would support the following
	94	input options:
	95
	96	- JSON in FEVER's aggregate format delivered via AMQP from a temporary queue
	97	attached to the exchange `tdh.pdns` on `localhost` port 5762, authenticated
	98	with user `guest` and password `guest`
73	99	- JSON in FEVER's aggregate format parsed from HTTP POST requests on port 8081 on the local system
74	100	- JSON in gopassivedns's format, fed into the UNIX socket `/tmp/balboa.sock` created by balboa
75	101
76		All of these feeders accept input simultaneously, there is no distinction made as to where an observation has come from. It is possible to specify multiple feeders of the same type but with different settings as long as their `name`s are unique.
	102	All of these feeders accept input simultaneously, there is no distinction made
	103	as to where an observation has come from. It is possible to specify multiple
	104	feeders of the same type but with different settings as long as their `name`s
	105	are unique.
77	106
78	107	### Configuring the database backend
79	108
80		Multiple database backends are supported to store pDNS observations persistently. The one to use in a particular instance can be configured separately in another YAML file (to be passed via the `-d` parameter to `balboa serve`). For example, to use the RocksDB backend storing data in `/tmp/balboa`, one would specify:
81
82		```yaml
83		database:
84		name: Local RocksDB
85		type: rocksdb
86		db_path: /tmp/balboa
87		```
88
89		To alternatively, for instance, send all data to a Cassandra cluster, use the following configuration:
90
91		```yaml
92		database:
93		name: Cassandra cluster
94		type: cassandra
95		hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]
96		username: cassandra
97		password: cassandra
98		```
99
100		which would use the specified nodes to access the cluster. Only one database can be configured at a time.
101
102		### Running the server and consuming input
103
104		All interaction with the service on the command line takes place via the `balboa` executable. The server can be started using:
105
106		```
107		$ balboa serve -l ''
108		INFO[0000] Local RocksDB: memory budget empty, using default of 128MB
109		INFO[0000] starting database Local RocksDB
110		INFO[0000] opening database...
111		INFO[0000] database ready
	109	Multiple database backends are supported to store pDNS observations
	110	persistently. Each database backend is provided as a self-contained binary
	111	(executable). The frontend connects to exactly one database backend. The
	112	backend, however, supports multiple client or frontend connections.
	113
	114	### Running the backend and frontend services, consuming input
	115
	116	All interaction with the frontend on the command line takes place via the
	117	`balboa` frontend executable. The frontend depends on a backend service. E.g
	118	the RocksDB backend can be started using:
	119
	120	```text
	121	$ balboa-rocksdb -h
	122	`balboa-rocksdb` provides a pdns database backend for `balboa`
	123
	124	Usage: balboa-rocksdb [options]
	125
	126	-h display help
	127	-D daemonize (default: off)
	128	-d <path> path to rocksdb database (default: `/tmp/balboa-rocksdb`)
	129	-l listen address (default: 127.0.0.1)
	130	-p listen port (default: 4242)
	131	-v increase verbosity; can be passed multiple times
	132	-j thread throttle limit, maximum concurrent connections (default: 64)
	133	--membudget <memory-in-bytes> rocksdb membudget option (value: 134217728)
	134	--parallelism <number-of-threads> rocksdb parallelism option (value: 8)
	135	--max_log_file_size <size> rocksdb log file size option (value: 10485760)
	136	--max_open_files <number> rocksdb max number of open files (value: 300)
	137	--keep_log_file_num <number> rocksdb max number of log files (value: 2)
	138	--database_path <path> same as `-d`
	139	--version show version thenp exit
	140
	141	$ balboa-rocksdb --database_path /data/pdns -l 127.0.0.1 -p 4242
	142	```
	143
	144	After starting the backend the `balboa` frontend can be started as follows:
	145
	146	```text
	147	$ balboa serve -l '' --host 127.0.0.1:4242
112	148	INFO[0000] starting feeder AMQPInput2
113	149	INFO[0000] starting feeder HTTP Input
114	150	INFO[0000] accepting submissions on port 8081
115	151	INFO[0000] starting feeder Socket Input
116	152	INFO[0000] starting feeder Suricata Socket Input
	153	INFO[0000] ConsumeFeed() starting
117	154	INFO[0000] serving GraphQL on port 8080
118		```
119
120		Depending on the size of the existing database, the start-up time ("opening database...") might be rather long if there is a lot of log to go through.
121
122		After startup, the feeders are free to be used for data ingest. For example, one might do some of the following to test data consumption (assuming the feeders above are used):
	155	...
	156	```
	157
	158	After startup, the feeders are free to be used for data ingest. For example,
	159	one might do some of the following to test data consumption (assuming the
	160	feeders above are used):
123	161
124	162	- for AMQP:
125
	163
	164	```text
	165	$ scripts/mkjson.py \| rabbitmqadmin publish routing_key="" exchange=tdh.pdns
	166	...
126	167	```
127		$ scripts/mkjson.py \| rabbitmqadmin publish routing_key="" exchange=tdh.pdns
	168
	169	- for HTTP:
	170
	171	```text
	172	$ scripts/mkjson.py \| curl -d@- -qs --header "X-Sensor-ID: abcde" http://localhost:8081/submit
	173	...
128	174	```
129
130		- for HTTP:
	175
	176	- for socket:
	177
	178	```text
	179	$ sudo gopassivedns -dev eth0 \| socat /tmp/balboa.sock STDIN
	180	...
131	181	```
132		$ scripts/mkjson.py \| curl -d@- -qs --header "X-Sensor-ID: abcde" http://localhost:8081/submit
133		```
134
135		- for socket:
136		```
137		$ sudo gopassivedns -dev eth0 \| socat /tmp/balboa.sock STDIN
138		```
139
140		Besides these asynchronous updates, it is always possible to use the `announceObservation` mutation in the GraphQL interface to explicitly add an observation, returning the updated data entry immediately:
141
142		```graphql
143		mutation {
144		announceObservation(observation: {
145		rrname: "test.foobar.de",
146		rrtype: A,
147		rdata: "1.2.3.4",
148		time_first: 1531943211,
149		time_last: 1531949570,
150		sensor_id: "abcde",
151		count: 3
152		}) {
153		count
154		}
155		}
156		```
157
158		This request would synchronously add a new observation with the given input data to the database and then return the new, updated `count` value.
159	182
160	183	### Querying the server
161	184
162		The intended main interface for interacting with the server is via GraphQL. For example, the query
	185	The intended main interface for interacting with the server is via GraphQL. For
	186	example, the query
163	187
164	188	```graphql
165	189	query {
166		entries(rrname: "test.foobar.de", sensor_id: "abcde") {
	190	entries(rrname: "test.foobar.de", sensor_id: "abcde", limit: 1) {
167	191	rrname
168	192	rrtype
169	193	rdata
170	194	time_first
171	195	time_last
172	196	sensor_id
173		count
	197	count
174	198	}
175	199	}
176	200	```

195	219	}
196	220	```
197	221
198		This also works with `rdata` as the query parameter, but at least one of `rrname` or `rdata` must be stated. If there is no `sensor_id` parameter, then all results will be returned regardless of where the DNS answer was observed. Use the `time_first_rfc3339` and `time_last_rfc3339` instead of `time_first` and `time_last`, respectively, to get human-readable timestamps.
	222	This also works with `rdata` as the query parameter, but at least one of
	223	`rrname` or `rdata` must be stated. If there is no `sensor_id` parameter, then
	224	all results will be returned regardless of where the DNS answer was observed.
	225	Use the `time_first_rfc3339` and `time_last_rfc3339` instead of `time_first`
	226	and `time_last`, respectively, to get human-readable timestamps.
199	227
200	228	### Aliases
201	229
202		Sometimes it is interesting to ask for all the domain names that resolve to the same IP address. For this reason, the GraphQL API supports a virtual `aliases` field that returns all Entries with RRType `A` or `AAAA` that share the same address in the Rdata field.
	230	Sometimes it is interesting to ask for all the domain names that resolve to the
	231	same IP address. For this reason, the GraphQL API supports a virtual `aliases`
	232	field that returns all Entries with RRType `A` or `AAAA` that share the same
	233	address in the Rdata field.
203	234
204	235	Example:
205	236

213	244	time_last_rfc3339
214	245	aliases {
215	246	rrname
216		time_first_rfc3339
217		time_last_rfc3339
218	247	}
219	248	}
220	249	}

252	281
253	282	### Bulk queries
254	283
255		There is also a shortcut tool to make 'bulk' querying easier. For example, to get all the information on the hosts in range 1.2.0.0/16 as observed by sensor `abcde`, one can use:
256
257		```
	284	There is also a shortcut tool to make 'bulk' querying easier. For example, to
	285	get all the information on the hosts in range 1.2.0.0/16 as observed by sensor
	286	`abcde`, one can use:
	287
	288	```text
258	289	$ balboa query --sensor abcde 1.2.0.0/16
259	290	{"count":6,"time_first":1531943211,"time_last":1531949570,"rrtype":"A","rrname":"test.foobar.de","rdata":"1.2.3.4","sensor_id":"abcde"}
260	291	{"count":1,"time_first":1531943215,"time_last":1531949530,"rrtype":"A","rrname":"baz.foobar.de","rdata":"1.2.3.7","sensor_id":"abcde"}
261	292	```
262		Note that this tool currently only does a lot of concurrent individual queries! To improve performance in these cases it might be worthwhile to allow for range queries on the server side as well in the future.
	293
	294	Note that this tool currently only does a lot of concurrent individual queries!
	295	To improve performance in these cases it might be worthwhile to allow for range
	296	queries on the server side as well in the future.
263	297
264	298	### Other tools
265	299
266		Run `balboa` without arguments to list available subcommands and get a short description of what they do.
	300	Run `balboa` without arguments to list available subcommands and get a short
	301	description of what they do.
	302
	303	See also `README.md` in the `backends` directory.
267	304
268	305	## Author/Contact
269	306

+147

-0

backend/.clang-format less more

	0	---
	1	Language: Cpp
	2	# BasedOnStyle: Google
	3	AccessModifierOffset: -1
	4	AlignAfterOpenBracket: AlwaysBreak
	5	AlignConsecutiveAssignments: false
	6	AlignConsecutiveDeclarations: false
	7	AlignEscapedNewlines: Left
	8	AlignOperands: true
	9	AlignTrailingComments: false
	10	AllowAllParametersOfDeclarationOnNextLine: true
	11	AllowShortBlocksOnASingleLine: true
	12	AllowShortCaseLabelsOnASingleLine: true
	13	AllowShortFunctionsOnASingleLine: Empty
	14	AllowShortIfStatementsOnASingleLine: true
	15	AllowShortLoopsOnASingleLine: true
	16	AlwaysBreakAfterDefinitionReturnType: None
	17	AlwaysBreakAfterReturnType: None
	18	AlwaysBreakBeforeMultilineStrings: true
	19	AlwaysBreakTemplateDeclarations: Yes
	20	BinPackArguments: false
	21	BinPackParameters: false
	22	BraceWrapping:
	23	AfterClass: false
	24	AfterControlStatement: false
	25	AfterEnum: false
	26	AfterFunction: false
	27	AfterNamespace: false
	28	AfterObjCDeclaration: false
	29	AfterStruct: false
	30	AfterUnion: false
	31	AfterExternBlock: false
	32	BeforeCatch: false
	33	BeforeElse: false
	34	IndentBraces: false
	35	SplitEmptyFunction: false
	36	SplitEmptyRecord: false
	37	SplitEmptyNamespace: false
	38	BreakBeforeBinaryOperators: NonAssignment
	39	BreakBeforeBraces: Attach
	40	BreakBeforeInheritanceComma: true
	41	BreakInheritanceList: BeforeComma
	42	BreakBeforeTernaryOperators: true
	43	BreakConstructorInitializersBeforeComma: true
	44	BreakConstructorInitializers: BeforeComma
	45	BreakAfterJavaFieldAnnotations: false
	46	BreakStringLiterals: true
	47	ColumnLimit: 80
	48	CommentPragmas: '^ IWYU pragma:'
	49	CompactNamespaces: true
	50	ConstructorInitializerAllOnOneLineOrOnePerLine: false
	51	ConstructorInitializerIndentWidth: 4
	52	ContinuationIndentWidth: 4
	53	Cpp11BracedListStyle: true
	54	DerivePointerAlignment: false
	55	DisableFormat: false
	56	ExperimentalAutoDetectBinPacking: false
	57	FixNamespaceComments: true
	58	ForEachMacros:
	59	- foreach
	60	- Q_FOREACH
	61	- BOOST_FOREACH
	62	IncludeBlocks: Preserve
	63	IncludeCategories:
	64	- Regex: '^<ext/.*\.h>'
	65	Priority: 2
	66	- Regex: '^<.*\.h>'
	67	Priority: 1
	68	- Regex: '^<.*'
	69	Priority: 2
	70	- Regex: '.*'
	71	Priority: 3
	72	IncludeIsMainRegex: '([-_](test\|unittest))?$'
	73	IndentCaseLabels: false
	74	IndentPPDirectives: None
	75	IndentWidth: 2
	76	IndentWrappedFunctionNames: true
	77	JavaScriptQuotes: Leave
	78	JavaScriptWrapImports: true
	79	KeepEmptyLinesAtTheStartOfBlocks: false
	80	MacroBlockBegin: ''
	81	MacroBlockEnd: ''
	82	MaxEmptyLinesToKeep: 1
	83	NamespaceIndentation: None
	84	ObjCBinPackProtocolList: Never
	85	ObjCBlockIndentWidth: 2
	86	ObjCSpaceAfterProperty: false
	87	ObjCSpaceBeforeProtocolList: true
	88	PenaltyBreakAssignment: 2
	89	PenaltyBreakBeforeFirstCallParameter: 1
	90	PenaltyBreakComment: 300
	91	PenaltyBreakFirstLessLess: 120
	92	PenaltyBreakString: 1000
	93	PenaltyBreakTemplateDeclaration: 10
	94	PenaltyExcessCharacter: 1000000
	95	PenaltyReturnTypeOnItsOwnLine: 200
	96	PointerAlignment: Left
	97	RawStringFormats:
	98	- Language: Cpp
	99	Delimiters:
	100	- cc
	101	- CC
	102	- cpp
	103	- Cpp
	104	- CPP
	105	- 'c++'
	106	- 'C++'
	107	CanonicalDelimiter: ''
	108	BasedOnStyle: google
	109	- Language: TextProto
	110	Delimiters:
	111	- pb
	112	- PB
	113	- proto
	114	- PROTO
	115	EnclosingFunctions:
	116	- EqualsProto
	117	- EquivToProto
	118	- PARSE_PARTIAL_TEXT_PROTO
	119	- PARSE_TEST_PROTO
	120	- PARSE_TEXT_PROTO
	121	- ParseTextOrDie
	122	- ParseTextProtoOrDie
	123	CanonicalDelimiter: ''
	124	BasedOnStyle: google
	125	ReflowComments: true
	126	SortIncludes: true
	127	SortUsingDeclarations: true
	128	SpaceAfterCStyleCast: false
	129	SpaceAfterTemplateKeyword: true
	130	SpaceBeforeAssignmentOperators: true
	131	SpaceBeforeCpp11BracedList: false
	132	SpaceBeforeCtorInitializerColon: false
	133	SpaceBeforeInheritanceColon: false
	134	SpaceBeforeParens: Never
	135	SpaceBeforeRangeBasedForLoopColon: false
	136	SpaceInEmptyParentheses: false
	137	SpacesBeforeTrailingComments: 1
	138	SpacesInAngles: false
	139	SpacesInContainerLiterals: false
	140	SpacesInCStyleCastParentheses: false
	141	SpacesInParentheses: false
	142	SpacesInSquareBrackets: false
	143	Standard: Cpp11
	144	TabWidth: 8
	145	UseTab: Never
	146	...

+30

-0

backend/Makefile less more

	0
	1	MAKEFLAGS+=--no-print-directory
	2
	3	all: rocksdb console mock
	4
	5	rocksdb:
	6	${MAKE} -C balboa-rocksdb
	7
	8	console:
	9	${MAKE} -C balboa-backend-console
	10
	11	mock:
	12	${MAKE} -C balboa-mock
	13
	14	sqlite:
	15	${MAKE} -C balboa-sqlite
	16
	17	style:
	18	clang-format -i \
	19	lib/protocol.{c,h} lib/engine.{c,h} lib/daemon.{c,h} lib/alloc.h lib/trace.{c,h} \
	20	balboa-rocksdb/rocksdb-impl.{c,h} balboa-rocksdb/main.c \
	21	balboa-mock/mock-impl.{c,h} balboa-mock/main.c \
	22	balboa-sqlite/sqlite-impl.{c,h} balboa-sqlite/main.c \
	23	balboa-backend-console/main.c
	24
	25	clean:
	26	${MAKE} -C balboa-mock clean
	27	${MAKE} -C balboa-backend-console clean
	28	${MAKE} -C balboa-rocksdb clean
	29	${MAKE} -C balboa-sqlite clean

+118

-0

backend/PROTOCOL.md less more

	0
	1	# `balboa` Frontend <-> Backend (ad-hoc) Protocol
	2
	3	All messges between the frontend and backend are encoded using the [msgpack][1]
	4	format. We use a double encoding of outer and inner messages to be
	5	able to add compression and authentication later on.
	6
	7	```text
	8	enum typed_message_id(int){
	9	PROTOCOL_INPUT_REQUEST=1
	10	PROTOCOL_QUERY_REQUEST=2
	11	PROTOCOL_BACKUP_REQUEST=3
	12	PROTOCOL_DUMP_REQUEST=4
	13	PROTOCOL_ERROR_RESPONSE=128
	14	PROTOCOL_QUERY_RESPONSE=129
	15	PROTOCOL_QUERY_STREAM_START_RESPONSE=130
	16	PROTOCOL_QUERY_STREAM_DATA_RESPONSE=131
	17	PROTOCOL_QUERY_STREAM_END_RESPONSE=132
	18	}
	19	// the typed outer message
	20	struct typed_message{
	21	type: typed_message_id where field="T"
	22	encoded_message: bytestring where field="M"
	23	}
	24	```
	25
	26	Depending on the value of the `type` field, the `encoded_message` field contains
	27	the actual inner [msgpack][1] encoded message (using msgpack encoded data in
	28	`encoded_message` is not mandatory, could be plain text json as well).
	29
	30	All messages are delivered in an asynchronous fashion and are not explicitly ack'ed.
	31
	32	[1]: https://msgpack.org/
	33
	34	# Inner Messages
	35
	36	## Input Request Message
	37
	38	```text
	39	struct pdns_entry{
	40	rrname: bytestring where field="N"
	41	rdata: bytestring where field="D"
	42	rrtype: bytestring where field="T"
	43	sensorid: bytestring where field="I"
	44	count: uint32 where field="C"
	45	first_seen: timestamp_in_seconds where field="F"
	46	last_seen: timestamp_in_seconds where field="L"
	47	}
	48	struct input_message{
	49	entry: pdns_entry where field="O"
	50	}
	51	```
	52
	53	## Query Request Message
	54
	55	```text
	56	struct query_request{
	57	qrrname: bytestring where field="Qrrname"
	58	have_rrname: bool where field="Hrrname"
	59	qrdata: bytestring where field="Qrdata"
	60	have_qrdata: bool where field="Hrdata"
	61	qrrtype: bytestring where field="Qrrtype"
	62	have_rrtype: bool where field="Hrrtype"
	63	qsensorid: bytestring where field="Qsensorid"
	64	have_sensorid: bool where field="Hsensorid"
	65	limit: int where field="Limit"
	66	}
	67	```
	68
	69	## Query Response Message
	70
	71	```text
	72	struct query_response{
	73	observations: array(pdns_entry) where field="O"
	74	}
	75	```
	76
	77	## Query Stream Response Start
	78
	79	```text
	80	struct query_stream_start_response{
	81	// empty
	82	}
	83	```
	84
	85	## Query Stream Response Data
	86
	87	```text
	88	struct query_stream_data_response{
	89	embed pdns_entry
	90	}
	91	```
	92
	93	## Query Stream Response End
	94
	95	```text
	96	struct query_stream_end_response{
	97	// empty
	98	}
	99	```
	100
	101	## Dump Request Message
	102
	103	```text
	104	struct dump_request{
	105	// currently unused
	106	path: bytestring where field="P"
	107	}
	108	```
	109
	110	## Backup Request Message
	111
	112	```text
	113	struct backup_request{
	114	// currently unused
	115	path: bytestring where field="P"
	116	}
	117	```

+184

-0

backend/README.md less more

	0	# balboa backends
	1
	2	This directory contains the code for database backends leveraged by
	3	[balboa](https://www.github.colm/DCSO/balboa)
	4
	5	## Building and Installation
	6
	7	### RocksDB Backend
	8
	9	First, install the required depencencies. Aside from `make` and a working `gcc`
	10	installation, `rocksdb` development files are necessary.
	11
	12	On Debian (testing and stretch-backports), one can satisfy these dependencies
	13	with:
	14
	15	```text
	16	% apt install librocksdb-dev
	17	...
	18	```
	19
	20	On Void Linux
	21
	22	```text
	23	% xbps-install rocksdb-devel
	24	...
	25	```
	26
	27	Building the RocksDB backend:
	28
	29	```text
	30	$ cd backend/balboa/balboa-rocksdb
	31	$ make
	32	...
	33	```
	34
	35	This yields the backend binary `build/linux/balboa-rocksdb`.
	36
	37	### Usage
	38
	39	Show available parameters to the RocksDB backend:
	40
	41	```text
	42	$ balboa-rocksdb -h
	43	`balboa-rocksdb` provides a pdns database backend for `balboa`
	44
	45	Usage: balboa-rocksdb [options]
	46
	47	-h display help
	48	-D daemonize (default: off)
	49	-d <path> path to rocksdb database (default: `/tmp/balboa-rocksdb`)
	50	-l listen address (default: 127.0.0.1)
	51	-p listen port (default: 4242)
	52	-v increase verbosity; can be passed multiple times
	53	-j thread throttle limit, maximum concurrent connections (default: 64)
	54	--membudget <memory-in-bytes> rocksdb membudget option (value: 134217728)
	55	--parallelism <number-of-threads> rocksdb parallelism option (value: 8)
	56	--max_log_file_size <size> rocksdb log file size option (value: 10485760)
	57	--max_open_files <number> rocksdb max number of open files (value: 300)
	58	--keep_log_file_num <number> rocksdb max number of log files (value: 2)
	59	--database_path <path> same as `-d`
	60	--version show version then exit
	61	```
	62
	63	Now start balboa and the backend to feed pDNS observations into it:
	64
	65	```text
	66	$ balboa-rocksdb -p 4242 -h 127.0.0.1 -d /tmp/balboa-rocksdb -D
	67	$ balboa serve -l '' -host 127.0.0.1:4242 -f my-feeders.yaml
	68	...
	69	```
	70
	71	### Other tools
	72
	73	#### balboa-backend-console
	74
	75	`balboa-backend-console` is a small utility managing balboa backends. It speaks
	76	the backend protocol directly. You can `backup`, `dump` and `replay`
	77	databases. Building is as easy as:
	78
	79	```text
	80	$ cd backend/balboa-backend-console
	81	$ make
	82	...
	83	```
	84
	85	assuming `make` and `gcc` are working on your system. It drops the self
	86	contained binary `build/linux/balboa-backend-console`.
	87
	88	```text
	89	$ balboa-backend-console -h
	90	`balboa-backend-console` is a management tool for `balboa-backends`
	91
	92	Usage: balboa-backend-console <--version\|help\|jsonize\|dump\|replay> [options]
	93
	94	Command help:
	95	show help
	96
	97	Command jsonize:
	98	read a dump file and print all entries as json
	99
	100	-d <path> path to the dump file to read
	101
	102	Command dump:
	103	connect to a `balboa-backend` and request a dump of all data to local stdout
	104
	105	-h <host> ip address of the `balboa-backend` (default: 127.0.0.1)
	106	-p <port> port of the `balboa-backend` (default: 4242)
	107	-v increase verbosity; can be passed multiple times
	108	-d <remote-dump-path> unused/ignored (default: -)
	109
	110	Command replay:
	111	replay a previously generated database dump
	112
	113	-d <path> database dump file or `-` for stdin (default: -)
	114	-h <host> ip address of the `balboa-backend` (default: 127.0.0.1)
	115	-p <port> port of the `balboa-backend` (default: 4242)
	116	-v increase verbosity; can be passed multiple times
	117
	118	Examples:
	119
	120	balboa-backend-console jsonize -r /tmp/pdns.dmp
	121	lz4cat /tmp/pdns.dmp.lz4 \| balboa-backend-console jsonize
	122	```
	123
	124	#### balboa-rocksdb-v1-dump
	125
	126	`balboa-rocksdb-v1-dump` is a small utility dealing with migration from
	127	balboa v1 databases to the new and internal format of v2.
	128
	129	```
	130	$ cd backend/balboa-rocksdb-v1-dump
	131	$ make
	132	...
	133	```
	134
	135	After a successful build, the binary is located at
	136	`build/linux/balboa-rocksdb-v1-dump`.
	137
	138	## Migrating from balboa/rocksdb v1
	139
	140	`balboa` v1 uses a different format to store data in the `rocksdb` backend. In
	141	order not to loose all of our precicous pDNS observations we need a migration
	142	strategy. The suggested approach looks a bit cumbersome but is in part due to
	143	the reason `rocksdb` does not yet support hot backup.
	144
	145	Let's assume you have a moderatly large pDNS observation database stored in
	146	`/data/balboa-rocksdb-v1`.
	147
	148	```text
	149	$ du -s /data/balboa-rocksdb-v1
	150	42GB
	151	```
	152
	153	Now, we want to migrate this database to balboa v2. The feeder configuration
	154	remains the same. Start the new balboa backend and frontend.
	155
	156	```text
	157	$ balboa-rocksdb -d /data/balboa-rocksdb-v2 -h 127.0.0.1 -p 4242
	158	$ balboa serve -l '' -f my-feeders.yaml -host 127.0.0.1:4242
	159	...
	160	```
	161
	162	Stop the old balboa v1 service and dump the database:
	163
	164	```text
	165	$ balboa-rocksdb-v1-dump dump /data/balboa-rocksdb \| lz4 > /data/pdns-backup.dmp.lz4
	166	$ lz4cat /data/pdns-backup.dmp.lz4 \| balboa-backend-console replay -h 127.0.0.1 -p 4242
	167	...
	168	```
	169
	170	`balboa-rocksdb-v1-dump` will dump the observation entries to stdout which in
	171	turn gets redirected to the `balboa-backend-console` tool in `replay` mode
	172	(reading from stdin in this case, use `-d <path>` to read from dump file)
	173
	174	Wait some time. Done.
	175
	176	## Author/Contact
	177
	178	- Sascha Steinbiss
	179	- Alexander Wamack
	180
	181	## License
	182
	183	BSD-3-clause

+52

-0

backend/balboa-backend-console/Makefile less more

	0
	1	CROSS_HOST?=$(shell uname -m)
	2	CROSS_PREFIX?=
	3	CCOMPILER?=gcc
	4
	5	OUT=build/
	6
	7	CFLAGS?=
	8	CFLAGS+=-pipe -static -s -Ofast -flto
	9
	10	ifeq ($(CCOMPILER),gcc)
	11	CFLAGS+=-fwhole-program -fmax-errors=3 -D__GCC__
	12	endif
	13
	14	ifeq ($(CCOMPILER),clang)
	15	CFLAGS+=-D__CLANG__
	16	endif
	17
	18	CFLAGS+=-std=c11 -Wall -Wextra -D_GNU_SOURCE -D__TRACE__ -DNDEBUG
	19	CFLAGS+=-I. -I../lib
	20	CFLAGS+=-DMPACK_HAS_CONFIG
	21	LDFLAGS?=
	22	LDFLAGS+=-pthread
	23
	24	MAKEFLAGS+=--no-print-directory
	25
	26	CC=$(CROSS_PREFIX)$(CCOMPILER)
	27
	28	hdr-lib=bs.h trace.h protocol.h engine.h mpack-config.h
	29	hdr-lib-y=$(addprefix ../lib/,$(hdr-lib))
	30
	31	src-console=mpack.c trace.c protocol.c engine.c
	32	src-console-y=$(addprefix ../lib/,$(src-console)) main.c
	33
	34	target-console-y=$(OUT)$(CROSS_PREFIX)balboa-backend-console
	35
	36	dirs-y=.
	37
	38	all: $(target-console-y)
	39
	40	$(OUT)build:
	41	@echo " mkdir"
	42	$(Q)mkdir -p $(addprefix $(OUT),$(dirs-y))
	43	$(Q)touch $@
	44
	45	$(target-console-y): Makefile $(OUT)build $(src-console-y) $(hdr-lib-y) Makefile
	46	$(CC) $(CFLAGS) $(src-console-y) -o $(target-console-y) $(LDFLAGS)
	47
	48	clean:
	49	rm -f $(target-console-y)
	50	rm -f $(OUT)build
	51	rmdir $(OUT)

+529

-0

backend/balboa-backend-console/main.c less more

	0	// balboa
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	#include <arpa/inet.h>
	4	#include <assert.h>
	5	#include <errno.h>
	6	#include <netdb.h>
	7	#include <netinet/in.h>
	8	#include <stdio.h>
	9	#include <stdlib.h>
	10	#include <string.h>
	11	#include <sys/socket.h>
	12	#include <sys/types.h>
	13	#include <unistd.h>
	14
	15	#include <bs.h>
	16	#include <engine.h>
	17	#include <ketopt.h>
	18	#include <protocol.h>
	19	#include <trace.h>
	20
	21	typedef struct state_t state_t;
	22	struct state_t {
	23	uint8_t* scrtch0;
	24	size_t scrtch0_sz;
	25	FILE* os;
	26	int sock;
	27	int (dump_entry_cb)(state_t state, protocol_entry_t* entry);
	28	};
	29
	30	static int dump_state_init(state_t* state) {
	31	state->scrtch0_sz = 1024 * 1024 * 10;
	32	state->scrtch0 = malloc(state->scrtch0_sz);
	33	if(state->scrtch0 == NULL) { return (-1); }
	34	state->os = NULL;
	35	state->sock = -1;
	36	return (0);
	37	}
	38
	39	static void dump_state_teardown(state_t* state) {
	40	free(state->scrtch0);
	41	}
	42
	43	static ssize_t dump_process(state_t* state, FILE* is) {
	44	protocol_dump_stream_t* stream = blb_protocol_dump_stream_new(is);
	45	ssize_t entries = 0;
	46	while(1) {
	47	protocol_entry_t entry;
	48	int rc = blb_protocol_dump_stream_decode(stream, &entry);
	49	switch(rc) {
	50	case 0: {
	51	int rc = state->dump_entry_cb(state, &entry);
	52	if(rc != 0) {
	53	L(log_error("dump_entry_cb() failed with `%d`", rc));
	54	return (-entries);
	55	}
	56	entries++;
	57	continue;
	58	}
	59	case -1: return (entries);
	60	default:
	61	L(log_error("blb_dump_stream_decode() failed with `%d`", rc));
	62	blb_protocol_dump_stream_teardown(stream);
	63	return (-entries);
	64	}
	65	}
	66	blb_protocol_dump_stream_teardown(stream);
	67	return (entries);
	68	}
	69
	70	static int dump(state_t* state, const char* dump_file) {
	71	ASSERT(state != NULL);
	72	ASSERT(state->dump_entry_cb != NULL);
	73	V(log_info("dump file is `%s`", dump_file));
	74	FILE* f = NULL;
	75	if(strcmp(dump_file, "-") == 0) {
	76	f = stdin;
	77	} else {
	78	f = fopen(dump_file, "rb");
	79	}
	80	if(f == NULL) {
	81	L(log_error("unable to open file `%s`", dump_file));
	82	return (-1);
	83	}
	84	ssize_t rc = dump_process(state, f);
	85	L(log_info("done; processed `%zd` entries", rc));
	86	dump_state_teardown(state);
	87	fclose(f);
	88	if(rc < 0) {
	89	return (-1);
	90	} else {
	91	return (0);
	92	}
	93	}
	94
	95	static void dump_entry_as_json(
	96	FILE* os, uint8_t* p, size_t p_sz, const protocol_entry_t* entry) {
	97	bytestring_sink_t __sink = bs_sink(p, p_sz);
	98	bytestring_sink_t* sink = &__sink;
	99	int ok = 0;
	100	char buf[64] = {0};
	101	ok += bs_cat(sink, "{\"rrname\":\"", 11);
	102	ok +=
	103	bs_append_escape(sink, (const uint8_t*)entry->rrname, entry->rrname_len);
	104	ok += bs_cat(sink, "\",\"rrtype\":\"", 12);
	105	ok +=
	106	bs_append_escape(sink, (const uint8_t*)entry->rrtype, entry->rrtype_len);
	107	ok += bs_cat(sink, "\",\"sensor_id\":\"", 15);
	108	ok += bs_append_escape(
	109	sink, (const uint8_t*)entry->sensorid, entry->sensorid_len);
	110	ok += bs_cat(sink, "\",\"rdata\":\"", 11);
	111	ok += bs_append_escape(sink, (const uint8_t*)entry->rdata, entry->rdata_len);
	112	ok += bs_cat(sink, "\",\"count\":", 10);
	113	snprintf(buf, 63, "%u", entry->count);
	114	ok += bs_cat(sink, buf, strlen(buf));
	115	ok += bs_cat(sink, ",\"first_seen\":", 14);
	116	snprintf(buf, 63, "%u", entry->first_seen);
	117	ok += bs_cat(sink, buf, strlen(buf));
	118	ok += bs_cat(sink, ",\"last_seen\":", 13);
	119	snprintf(buf, 63, "%u", entry->last_seen);
	120	ok += bs_cat(sink, buf, strlen(buf));
	121	ok += bs_append1(sink, '}');
	122	ok += bs_append1(sink, '\n');
	123	if(ok == 0) {
	124	fwrite(sink->p, sink->index, 1, os);
	125	} else {
	126	fputs("{\"error\":\"buffer-out-of-space\"}", os);
	127	}
	128	}
	129
	130	static int dump_entry_json_cb(state_t* state, protocol_entry_t* entry) {
	131	ASSERT(state->os != NULL);
	132	dump_entry_as_json(state->os, state->scrtch0, state->scrtch0_sz, entry);
	133	return (0);
	134	}
	135
	136	static int dump_entry_replay_cb(state_t* state, protocol_entry_t* entry) {
	137	ASSERT(state->sock != -1);
	138
	139	protocol_input_request_t input = {.entry = *entry};
	140	ssize_t rc = blb_protocol_encode_input_request(
	141	&input, (char*)state->scrtch0, state->scrtch0_sz);
	142	if(rc <= 0) {
	143	L(log_error("unable to encode input request"));
	144	return (-1);
	145	}
	146
	147	uint8_t* p = state->scrtch0;
	148	ssize_t r = rc;
	149	while(r > 0) {
	150	ssize_t rc = write(state->sock, p, r);
	151	if(rc < 0) {
	152	L(log_error("write() failed with `%s`", strerror(errno)));
	153	return (-1);
	154	} else if(rc == 0 && errno == EINTR) {
	155	continue;
	156	}
	157	r -= rc;
	158	p += rc;
	159	}
	160	return (0);
	161	}
	162
	163	static int main_query(int argc, char** argv) {
	164	engine_config_t engine_config = blb_engine_client_config_init();
	165	trace_config_t trace_config = {.stream = stderr,
	166	.host = "pdns",
	167	.app = "balboa-backend-console",
	168	// leaking process number ...
	169	.procid = getpid()};
	170	protocol_query_request_t __query = {0}, *query = &__query;
	171	query->limit = 100;
	172	ketopt_t opt = KETOPT_INIT;
	173	int c;
	174	while((c = ketopt(&opt, argc, argv, 1, "h:p:r:d:s:l:vSR", NULL)) >= 0) {
	175	switch(c) {
	176	case 'v': trace_config.verbosity += 1; break;
	177	case 'h': engine_config.host = opt.arg; break;
	178	case 'p': engine_config.port = atoi(opt.arg); break;
	179	case 'd':
	180	if(opt.arg == NULL) {
	181	L(log_emergency("string for option `-d` required"));
	182	}
	183	query->qrdata = opt.arg;
	184	query->qrdata_len = strlen(opt.arg);
	185	break;
	186	case 'r':
	187	if(opt.arg == NULL) {
	188	L(log_emergency("string for option `-r` required"));
	189	}
	190	query->qrrname = opt.arg;
	191	query->qrrname_len = strlen(opt.arg);
	192	break;
	193	case 's':
	194	if(opt.arg == NULL) {
	195	L(log_emergency("string for option `-s` required"));
	196	}
	197	query->qsensorid = opt.arg;
	198	query->qsensorid_len = strlen(opt.arg);
	199	break;
	200	default: break;
	201	}
	202	}
	203
	204	theTrace_stream_use(&trace_config);
	205
	206	conn_t* conn = blb_engine_client_new(&engine_config);
	207	engine_t* engine = conn->engine;
	208
	209	V(blb_protocol_log_query(query));
	210
	211	ssize_t used = blb_protocol_encode_query_request(
	212	query, conn->scrtch, ENGINE_CONN_SCRTCH_SZ);
	213	if(used <= 0) {
	214	L(log_error("unable to encode query"));
	215	blb_engine_teardown(engine);
	216	blb_engine_conn_teardown(conn);
	217	return (-1);
	218	}
	219
	220	int rc = blb_conn_write_all(conn, conn->scrtch, used);
	221	if(rc != 0) {
	222	L(log_debug("blb_conn_write_all() failed"));
	223	blb_engine_teardown(engine);
	224	blb_engine_conn_teardown(conn);
	225	return (-1);
	226	}
	227
	228	protocol_stream_t* stream = blb_engine_stream_new(conn);
	229	if(stream == NULL) {
	230	L(log_error("blb_engine_stream_new() failed"));
	231	blb_engine_teardown(engine);
	232	blb_engine_conn_teardown(conn);
	233	return (-1);
	234	}
	235
	236	enum state_t { START, STREAM, END };
	237
	238	enum state_t st = START;
	239	while(1) {
	240	protocol_message_t msg;
	241	int rc = blb_protocol_stream_decode(stream, &msg);
	242	if(rc < -1) {
	243	L(log_error("blb_protocol_stream_decode() failed"));
	244	blb_protocol_stream_teardown(stream);
	245	blb_engine_teardown(engine);
	246	blb_engine_conn_teardown(conn);
	247	return (-1);
	248	} else if(rc == -1) {
	249	break;
	250	}
	251	switch(st) {
	252	case START: {
	253	switch(msg.ty) {
	254	case PROTOCOL_QUERY_STREAM_START_RESPONSE: st = STREAM; break;
	255	default: L(log_emergency("(start) received invalid message"));
	256	}
	257	break;
	258	}
	259	case STREAM: {
	260	switch(msg.ty) {
	261	case PROTOCOL_QUERY_STREAM_END_RESPONSE: st = END; goto done;
	262	case PROTOCOL_QUERY_STREAM_DATA_RESPONSE: {
	263	uint8_t json[1024 * 10];
	264	dump_entry_as_json(stdout, json, sizeof(json), &msg.u.entry);
	265	break;
	266	}
	267	default: L(log_emergency("(stream) received invalid message"));
	268	}
	269	break;
	270	}
	271	default: L(log_emergency("invalid state: impossible"));
	272	}
	273	}
	274	done:
	275	blb_protocol_stream_teardown(stream);
	276	blb_engine_teardown(engine);
	277	blb_engine_conn_teardown(conn);
	278	return (0);
	279	}
	280
	281	static int main_jsonize(int argc, char** argv) {
	282	const char* dump_file = "-";
	283	int verbosity = 0;
	284	trace_config_t trace_config = {.stream = stderr,
	285	.host = "pdns",
	286	.app = "balboa-backend-console",
	287	// leaking process number ...
	288	.procid = getpid()};
	289
	290	ketopt_t opt = KETOPT_INIT;
	291	int c;
	292	while((c = ketopt(&opt, argc, argv, 1, "d:v", NULL)) >= 0) {
	293	switch(c) {
	294	case 'd': dump_file = opt.arg; break;
	295	case 'v': verbosity += 1; break;
	296	default: break;
	297	}
	298	}
	299
	300	theTrace_stream_use(&trace_config);
	301	theTrace_set_verbosity(verbosity);
	302
	303	V(log_info("dump file is `%s`", dump_file));
	304
	305	state_t __state = {0}, *state = &__state;
	306	int state_ok = dump_state_init(state);
	307	if(state_ok != 0) {
	308	L(log_error("unable to initialize the dump state"));
	309	return (-1);
	310	}
	311	state->os = stdout;
	312	state->dump_entry_cb = dump_entry_json_cb;
	313	int rc = dump(state, dump_file);
	314	return (rc);
	315	}
	316
	317	static int dump_connect(const char* host, const char* _port) {
	318	int port = atoi(_port);
	319	struct sockaddr_in addr;
	320	int addr_ok = inet_pton(AF_INET, host, &addr.sin_addr);
	321	if(addr_ok != 1) { return (-1); }
	322	addr.sin_family = AF_INET;
	323	addr.sin_port = htons((uint16_t)port);
	324	int fd = socket(addr.sin_family, SOCK_STREAM, 0);
	325	if(fd < 0) { return (-1); }
	326	int rc = connect(fd, &addr, sizeof(struct sockaddr_in));
	327	if(rc < 0) {
	328	close(fd);
	329	return (-1);
	330	}
	331	return (fd);
	332	}
	333
	334	__attribute__((noreturn)) void version(void) {
	335	fprintf(stderr, "balboa-backend-console v2.0.0\n");
	336	exit(1);
	337	}
	338
	339	__attribute__((noreturn)) void usage(void) {
	340	fprintf(
	341	stderr,
	342	"\
	343	`balboa-backend-console` is a management tool for `balboa-backends`\n\
	344	\n\
	345	Usage: balboa-backend-console <--version\|help\|jsonize\|dump\|replay> [options]\n\
	346	\n\
	347	Command help:\n\
	348	show help\n\
	349	\n\
	350	Command jsonize:\n\
	351	read a dump file and print all entries as json\n\
	352	\n\
	353	-d <path> path to the dump file to read\n\
	354	\n\
	355	Command dump:\n\
	356	connect to a `balboa-backend` and request a dump of all data to local stdout\n\
	357	\n\
	358	-h <host> ip address of the `balboa-backend` (default: 127.0.0.1)\n\
	359	-p <port> port of the `balboa-backend` (default: 4242)\n\
	360	-v increase verbosity; can be passed multiple times\n\
	361	-d <remote-dump-path> unused/ignored (default: -)\n\
	362	\n\
	363	Command replay:\n\
	364	replay a previously generated database dump\n\
	365	\n\
	366	-d <path> database dump file or `-` for stdin (default: -)\n\
	367	-h <host> ip address of the `balboa-backend` (default: 127.0.0.1)\n\
	368	-p <port> port of the `balboa-backend` (default: 4242)\n\
	369	-v increase verbosity; can be passed multiple times\n\
	370	\n\
	371	Examples:\n\
	372	\n\
	373	balboa-backend-console jsonize -r /tmp/pdns.dmp\n\
	374	lz4cat /tmp/pdns.dmp.lz4 \| balboa-backend-console jsonize\n\
	375	\n");
	376	exit(1);
	377	}
	378
	379	static int main_dump(int argc, char** argv) {
	380	const char* host = "127.0.0.1";
	381	const char* port = "4242";
	382	const char* dump_path_hint = "-";
	383	int verbosity = 0;
	384	trace_config_t trace_config = {.stream = stderr,
	385	.host = "pdns",
	386	.app = "balboa-backend-console",
	387	// leaking process number ...
	388	.procid = getpid()};
	389	ketopt_t opt = KETOPT_INIT;
	390	int c;
	391	while((c = ketopt(&opt, argc, argv, 1, "h:p:d:v", NULL)) >= 0) {
	392	switch(c) {
	393	case 'h': host = opt.arg; break;
	394	case 'p': port = opt.arg; break;
	395	case 'v': verbosity += 1; break;
	396	case 'd': dump_path_hint = opt.arg; break;
	397	default: break;
	398	}
	399	}
	400
	401	theTrace_stream_use(&trace_config);
	402	theTrace_set_verbosity(verbosity);
	403
	404	V(log_info(
	405	"host `%s` port `%s` dump_path_hint `%s`", host, port, dump_path_hint));
	406	int sock = dump_connect(host, port);
	407	if(sock < 0) {
	408	L(log_error("unable to connect to backend"));
	409	return (-1);
	410	}
	411
	412	char scrtch[1024];
	413	size_t scrtch_sz = sizeof(scrtch);
	414	protocol_dump_request_t req = {.path = dump_path_hint};
	415	ssize_t used = blb_protocol_encode_dump_request(&req, scrtch, scrtch_sz);
	416	if(used <= 0) {
	417	L(log_error("blb_protocol_encode_dump_request() failed `%zd`", used));
	418	close(sock);
	419	return (-1);
	420	}
	421	char* p = scrtch;
	422	ssize_t r = used;
	423	while(r > 0) {
	424	ssize_t rc = write(sock, p, r);
	425	if(rc < 0) {
	426	L(log_error("write() failed with `%s`", strerror(errno)));
	427	close(sock);
	428	return (-1);
	429	} else if(rc == 0 && errno == EINTR) {
	430	continue;
	431	}
	432	r -= rc;
	433	p += rc;
	434	}
	435
	436	while(1) {
	437	ssize_t rc = read(sock, scrtch, scrtch_sz);
	438	if(rc == 0) {
	439	fflush(stdout);
	440	close(sock);
	441	return (0);
	442	} else if(rc < 0) {
	443	L(log_error("read() failed with `%s`", strerror(errno)));
	444	close(sock);
	445	return (-1);
	446	}
	447	rc = fwrite(scrtch, rc, 1, stdout);
	448	if(rc != 1) {
	449	L(log_error("fwrite() failed with `%s`", strerror(errno)));
	450	close(sock);
	451	return (-1);
	452	}
	453	}
	454	}
	455
	456	static int main_replay(int argc, char** argv) {
	457	const char* host = "127.0.0.1";
	458	const char* port = "4242";
	459	const char* dump_file = "-";
	460	int verbosity = 0;
	461	trace_config_t trace_config = {.stream = stderr,
	462	.host = "pdns",
	463	.app = "balboa-backend-console",
	464	// leaking process number ...
	465	.procid = getpid()};
	466
	467	ketopt_t opt = KETOPT_INIT;
	468	int c;
	469	while((c = ketopt(&opt, argc, argv, 1, "d:h:p:v", NULL)) >= 0) {
	470	switch(c) {
	471	case 'd': dump_file = opt.arg; break;
	472	case 'h': host = opt.arg; break;
	473	case 'p': port = opt.arg; break;
	474	case 'v': verbosity += 1; break;
	475	default: break;
	476	}
	477	}
	478
	479	theTrace_stream_use(&trace_config);
	480	theTrace_set_verbosity(verbosity);
	481
	482	V(log_info("host `%s` port `%s` dump_file `%s`", host, port, dump_file));
	483
	484	int sock = dump_connect(host, port);
	485	if(sock < 0) {
	486	L(log_error("unable to connect to backend"));
	487	return (-1);
	488	}
	489	state_t __state = {0}, *state = &__state;
	490	int state_ok = dump_state_init(state);
	491	if(state_ok != 0) {
	492	L(log_error("unable to initialize the dump state"));
	493	return (-1);
	494	}
	495	state->sock = sock;
	496	state->dump_entry_cb = dump_entry_replay_cb;
	497	int rc = dump(state, dump_file);
	498	return (rc);
	499	}
	500
	501	int main(int argc, char** argv) {
	502	int res = -1;
	503	if(argc < 2) {
	504	usage();
	505	} else if(strcmp(argv[1], "jsonize") == 0) {
	506	argc--;
	507	argv++;
	508	res = main_jsonize(argc, argv);
	509	} else if(strcmp(argv[1], "replay") == 0) {
	510	argc--;
	511	argv++;
	512	res = main_replay(argc, argv);
	513	} else if(strcmp(argv[1], "dump") == 0) {
	514	argc--;
	515	argv++;
	516	res = main_dump(argc, argv);
	517	} else if(strcmp(argv[1], "query") == 0) {
	518	argc--;
	519	argv++;
	520	res = main_query(argc, argv);
	521	} else if(strcmp(argv[1], "--version") == 0) {
	522	version();
	523	} else {
	524	usage();
	525	}
	526
	527	return (res);
	528	}⏎

+53

-0

backend/balboa-mock/Makefile less more

	0
	1	CROSS_HOST?=$(shell uname -m)
	2	CROSS_PREFIX?=
	3	CCOMPILER?=gcc
	4
	5	OUT=build/
	6
	7	CFLAGS?=
	8	CFLAGS+=-pipe -static -s -Ofast -flto
	9
	10	ifeq ($(CCOMPILER),gcc)
	11	CFLAGS+=-fwhole-program -fmax-errors=3 -D__GCC__
	12	endif
	13
	14	ifeq ($(CCOMPILER),clang)
	15	CFLAGS+=-D__CLANG__
	16	endif
	17
	18	CFLAGS+=-std=c11 -Wall -Wextra -D_GNU_SOURCE -D__TRACE__ -DNDEBUG
	19	CFLAGS+=-I. -I../lib
	20	CFLAGS+=-DMPACK_HAS_CONFIG
	21	LDFLAGS?=
	22	LDFLAGS+=-pthread
	23
	24	MAKEFLAGS+=--no-print-directory
	25
	26	CC=$(CROSS_PREFIX)$(CCOMPILER)
	27
	28	hdr-balboa-mock=engine.h trace.h daemon.h
	29	hdr-balboa-mock-y=$(addprefix ../lib/,$(hdr-balboa-mock)) mock-impl.h mpack-config.h
	30
	31	src-balboa-mock=trace.c daemon.c protocol.c mpack.c engine.c
	32	src-balboa-mock-y=$(addprefix ../lib/,$(src-balboa-mock))
	33	src-balboa-mock-y+=mock-impl.c main.c
	34
	35	target-balboa-mock-y=$(OUT)$(CROSS_PREFIX)balboa-mock
	36
	37	dirs-y=.
	38
	39	all: $(target-balboa-mock-y)
	40
	41	$(OUT)build:
	42	@echo " mkdir"
	43	$(Q)mkdir -p $(addprefix $(OUT),$(dirs-y))
	44	$(Q)touch $@
	45
	46	$(target-balboa-mock-y): $(OUT)build $(src-balboa-mock-y) $(hdr-balboa-mock-y) Makefile
	47	$(CC) $(CFLAGS) $(src-balboa-mock-y) -o $(target-balboa-mock-y) $(LDFLAGS)
	48
	49	clean:
	50	rm -f $(target-balboa-mock-y)
	51	rm -f $(OUT)build
	52	rmdir $(OUT)

+58

-0

backend/balboa-mock/main.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <engine.h>
	4	#include <ketopt.h>
	5	#include <mock-impl.h>
	6	#include <trace.h>
	7	#include <unistd.h>
	8
	9	int main(int argc, char** argv) {
	10	int daemonize = 0;
	11	engine_config_t engine_config = blb_engine_server_config_init();
	12	trace_config_t trace_config = {.stream = stderr,
	13	.host = "pdns",
	14	.app = argv[0],
	15	// leaking process number ...
	16	.procid = getpid(),
	17	.verbosity = 0};
	18	ketopt_t opt = KETOPT_INIT;
	19	int c;
	20	while((c = ketopt(&opt, argc, argv, 1, "j:l:p:vDSR", NULL)) >= 0) {
	21	switch(c) {
	22	case 'D': daemonize = 1; break;
	23	case 'l': engine_config.host = opt.arg; break;
	24	case 'p': engine_config.port = atoi(opt.arg); break;
	25	case 'v': trace_config.verbosity += 1; break;
	26	case 'j': engine_config.conn_throttle_limit = atoi(opt.arg); break;
	27	case 'S': engine_config.enable_signal_consumer = false; break;
	28	case 'R': engine_config.enable_stats_reporter = false; break;
	29	default: break;
	30	}
	31	}
	32
	33	theTrace_stream_use(&trace_config);
	34	if(daemonize
	35	&& (trace_config.stream == stderr \|\| trace_config.stream == stdout)) {
	36	theTrace_set_verbosity(0);
	37	}
	38
	39	db_t* db = blb_mock_open();
	40	if(db == NULL) { return (1); }
	41
	42	engine_config.db = db;
	43	engine_t* e = blb_engine_server_new(&engine_config);
	44	if(e == NULL) {
	45	L(log_error("unable to create engine"));
	46	blb_dbi_teardown(db);
	47	return (1);
	48	}
	49
	50	blb_engine_run(e);
	51
	52	blb_dbi_teardown(db);
	53
	54	blb_engine_teardown(e);
	55
	56	return (0);
	57	}⏎

+106

-0

backend/balboa-mock/mock-impl.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <stdio.h>
	4	#include <stdlib.h>
	5	#include <string.h>
	6
	7	#include <mock-impl.h>
	8
	9	typedef struct blb_mock_t blb_mock_t;
	10
	11	static void blb_mock_teardown(db_t* _db);
	12	static db_t* blb_mock_conn_init(conn_t* th, db_t* db);
	13	static void blb_mock_conn_deinit(conn_t* th, db_t* db);
	14	static int blb_mock_query(conn_t* th, const protocol_query_request_t* q);
	15	static int blb_mock_input(conn_t* th, const protocol_input_request_t* i);
	16	static void blb_mock_dump(conn_t* th, const protocol_dump_request_t* d);
	17	static void blb_mock_backup(conn_t* th, const protocol_backup_request_t* b);
	18
	19	static const dbi_t blb_mock_dbi = {.thread_init = blb_mock_conn_init,
	20	.thread_deinit = blb_mock_conn_deinit,
	21	.teardown = blb_mock_teardown,
	22	.query = blb_mock_query,
	23	.input = blb_mock_input,
	24	.backup = blb_mock_backup,
	25	.dump = blb_mock_dump};
	26
	27	struct blb_mock_t {
	28	const dbi_t* dbi;
	29	};
	30
	31	db_t* blb_mock_conn_init(conn_t* th, db_t* db) {
	32	(void)th;
	33	return (db);
	34	}
	35
	36	void blb_mock_conn_deinit(conn_t* th, db_t* db) {
	37	(void)th;
	38	(void)db;
	39	}
	40
	41	void blb_mock_teardown(db_t* _db) {
	42	ASSERT(_db->dbi == &blb_mock_dbi);
	43	blb_free(_db);
	44	}
	45
	46	static int blb_mock_query(conn_t* th, const protocol_query_request_t* q) {
	47	(void)q;
	48
	49	int start_ok = blb_conn_query_stream_start_response(th);
	50	if(start_ok != 0) {
	51	L(log_error("unable to start query stream response"));
	52	return (-1);
	53	}
	54
	55	protocol_entry_t __e, *e = &__e;
	56	e->sensorid = "test-sensor-id";
	57	e->sensorid_len = strlen(e->sensorid);
	58	e->rdata = "";
	59	e->rdata_len = 0;
	60	e->rrname = "test-rrname";
	61	e->rrname_len = strlen(e->rrname);
	62	e->rrtype = "A";
	63	e->rrtype_len = 1;
	64	e->count = 23;
	65	e->first_seen = 15000000;
	66	e->last_seen = 15001000;
	67	int push_ok = blb_conn_query_stream_push_response(th, e);
	68	if(push_ok != 0) {
	69	L(log_error("unable to push query response entry"));
	70	return (-1);
	71	}
	72
	73	(void)blb_conn_query_stream_end_response(th);
	74
	75	return (0);
	76	}
	77
	78	static int blb_mock_input(conn_t* th, const protocol_input_request_t* i) {
	79	ASSERT(th->db->dbi == &blb_mock_dbi);
	80
	81	T(blb_protocol_log_entry(&i->entry));
	82
	83	return (0);
	84	}
	85
	86	static void blb_mock_backup(conn_t* th, const protocol_backup_request_t* b) {
	87	ASSERT(th->db->dbi == &blb_mock_dbi);
	88	// blb_mock_t* db=(blb_mock_t*)th->db;
	89
	90	T(log_debug("backup `%.*s`", (int)b->path_len, b->path));
	91	}
	92
	93	static void blb_mock_dump(conn_t* th, const protocol_dump_request_t* d) {
	94	ASSERT(th->db->dbi == &blb_mock_dbi);
	95	// blb_mock_t* db=(blb_mock_t*)th->db;
	96
	97	T(log_debug("dump `%.*s`", (int)d->path_len, d->path));
	98	}
	99
	100	db_t* blb_mock_open() {
	101	blb_mock_t* db = blb_new(blb_mock_t);
	102	if(db == NULL) { return (NULL); }
	103	db->dbi = &blb_mock_dbi;
	104	return ((db_t*)db);
	105	}

+16

-0

backend/balboa-mock/mock-impl.h less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#ifndef __MOCK_H
	4	#define __MOCK_H
	5
	6	#include <stdbool.h>
	7	#include <stdint.h>
	8	#include <stdlib.h>
	9	#include <time.h>
	10
	11	#include <engine.h>
	12
	13	db_t* blb_mock_open(void);
	14
	15	#endif

+375

-0

backend/balboa-mock/mpack-config.h less more

	0	/**
	1	* @defgroup config Configuration Options
	2	*
	3	* Defines the MPack configuration options. You can configure MPack by
	4	* pre-defining any of the below options in your build system or project
	5	* settings.
	6	*
	7	* Custom configuration of MPack is not usually necessary. In almost all
	8	* cases you can ignore this and use the defaults. If you are using the
	9	* amalgamation package, you do not need to add @c mpack-defaults.h to your
	10	* project.
	11	*
	12	* If you do want to configure MPack, the easiest way is to pre-define some of
	13	* the below options as part of your build system or project settings. This
	14	* will override the below defaults.
	15	*
	16	* If you'd like to use a file for configuration instead, define
	17	* @ref MPACK_HAS_CONFIG to 1 in your build system or project settings.
	18	* This will cause MPack to include a file you create called @c mpack-config.h.
	19	* You can copy @c mpack-defaults.h to @c mpack-config.h and make your
	20	* changes, or create a blank @c mpack-config.h and set only the options you
	21	* want. The below settings are the defaults if they are not set by your
	22	* configuration file.
	23	*
	24	* @warning The value of all configuration options must be the same in
	25	* all translation units of your project, as well as in @c mpack.c itself.
	26	* These configuration options affect the layout of structs, among other
	27	* things, which cannot be different in source files that are linked
	28	* together.
	29	*
	30	* @{
	31	*/
	32
	33
	34	/**
	35	* @name Features
	36	* @{
	37	*/
	38
	39	#include <engine.h>
	40
	41	#define MPACK_MALLOC blb_malloc
	42	#define MPACK_REALLOC blb_realloc
	43	#define MPACK_FREE blb_free
	44
	45	/**
	46	* @def MPACK_READER
	47	*
	48	* Enables compilation of the base Tag Reader.
	49	*/
	50	#ifndef MPACK_READER
	51	#define MPACK_READER 1
	52	#endif
	53
	54	/**
	55	* @def MPACK_EXPECT
	56	*
	57	* Enables compilation of the static Expect API.
	58	*/
	59	#ifndef MPACK_EXPECT
	60	#define MPACK_EXPECT 1
	61	#endif
	62
	63	/**
	64	* @def MPACK_NODE
	65	*
	66	* Enables compilation of the dynamic Node API.
	67	*/
	68	#ifndef MPACK_NODE
	69	#define MPACK_NODE 1
	70	#endif
	71
	72	/**
	73	* @def MPACK_WRITER
	74	*
	75	* Enables compilation of the Writer.
	76	*/
	77	#ifndef MPACK_WRITER
	78	#define MPACK_WRITER 1
	79	#endif
	80
	81	/**
	82	* @def MPACK_COMPATIBILITY
	83	*
	84	* Enables compatibility features for reading and writing older
	85	* versions of MessagePack.
	86	*
	87	* This is disabled by default. When disabled, the behaviour is equivalent to
	88	* using the default version, @ref mpack_version_current.
	89	*
	90	* Enable this if you need to interoperate with applications or data that do
	91	* not support the new (v5) MessagePack spec. See the section on v4
	92	* compatibility in @ref docs/protocol.md for more information.
	93	*/
	94	#ifndef MPACK_COMPATIBILITY
	95	#define MPACK_COMPATIBILITY 0
	96	#endif
	97
	98	/**
	99	* @def MPACK_EXTENSIONS
	100	*
	101	* Enables the use of extension types.
	102	*
	103	* This is disabled by default. Define it to 1 to enable it. If disabled,
	104	* functions to read and write extensions will not exist, and any occurrence of
	105	* extension types in parsed messages will flag @ref mpack_error_invalid.
	106	*
	107	* MPack discourages the use of extension types. See the section on extension
	108	* types in @ref docs/protocol.md for more information.
	109	*/
	110	#ifndef MPACK_EXTENSIONS
	111	#define MPACK_EXTENSIONS 1
	112	#endif
	113
	114
	115	/**
	116	* @}
	117	*/
	118
	119
	120	/**
	121	* @name Dependencies
	122	* @{
	123	*/
	124
	125	/**
	126	* @def MPACK_HAS_CONFIG
	127	*
	128	* Enables the use of an @c mpack-config.h configuration file for MPack.
	129	* This file must be in the same folder as @c mpack.h, or it must be
	130	* available from your project's include paths.
	131	*/
	132	// This goes in your project settings.
	133
	134	/**
	135	* @def MPACK_STDLIB
	136	*
	137	* Enables the use of C stdlib. This allows the library to use malloc
	138	* for debugging and in allocation helpers.
	139	*/
	140	#ifndef MPACK_STDLIB
	141	#define MPACK_STDLIB 1
	142	#endif
	143
	144	/**
	145	* @def MPACK_STDIO
	146	*
	147	* Enables the use of C stdio. This adds helpers for easily
	148	* reading/writing C files and makes debugging easier.
	149	*/
	150	#ifndef MPACK_STDIO
	151	#define MPACK_STDIO 1
	152	#endif
	153
	154	/**
	155	* @}
	156	*/
	157
	158
	159	/**
	160	* @name System Functions
	161	* @{
	162	*/
	163
	164	/**
	165	* @def MPACK_MALLOC
	166	*
	167	* Defines the memory allocation function used by MPack. This is used by
	168	* helpers for automatically allocating data the correct size, and for
	169	* debugging functions. If this macro is undefined, the allocation helpers
	170	* will not be compiled.
	171	*
	172	* The default is @c malloc() if @ref MPACK_STDLIB is enabled.
	173	*/
	174	/**
	175	* @def MPACK_FREE
	176	*
	177	* Defines the memory free function used by MPack. This is used by helpers
	178	* for automatically allocating data the correct size. If this macro is
	179	* undefined, the allocation helpers will not be compiled.
	180	*
	181	* The default is @c free() if @ref MPACK_MALLOC has not been customized and
	182	* @ref MPACK_STDLIB is enabled.
	183	*/
	184	/**
	185	* @def MPACK_REALLOC
	186	*
	187	* Defines the realloc function used by MPack. It is used by growable
	188	* buffers to resize more efficiently.
	189	*
	190	* The default is @c realloc() if @ref MPACK_MALLOC has not been customized and
	191	* @ref MPACK_STDLIB is enabled.
	192	*
	193	* This is optional, even when @ref MPACK_MALLOC is used. If @ref MPACK_MALLOC is
	194	* set and @ref MPACK_REALLOC is not, @ref MPACK_MALLOC is used with a simple copy
	195	* to grow buffers.
	196	*/
	197	#if defined(MPACK_STDLIB) && MPACK_STDLIB && !defined(MPACK_MALLOC)
	198	#define MPACK_MALLOC malloc
	199	#define MPACK_REALLOC realloc
	200	#define MPACK_FREE free
	201	#endif
	202
	203	/**
	204	* @}
	205	*/
	206
	207
	208	/**
	209	* @name Debugging Options
	210	*/
	211
	212	/**
	213	* @def MPACK_DEBUG
	214	*
	215	* Enables debug features. You may want to wrap this around your
	216	* own debug preprocs. By default, this is enabled if @c DEBUG or @c _DEBUG
	217	* are defined. (@c NDEBUG is not used since it is allowed to have
	218	* different values in different translation units.)
	219	*/
	220	#if !defined(MPACK_DEBUG) && (defined(DEBUG) \|\| defined(_DEBUG))
	221	#define MPACK_DEBUG 1
	222	#endif
	223
	224	/**
	225	* @def MPACK_STRINGS
	226	*
	227	* Enables descriptive error and type strings.
	228	*
	229	* This can be turned off (by defining it to 0) to maximize space savings
	230	* on embedded devices. If this is disabled, string functions such as
	231	* mpack_error_to_string() and mpack_type_to_string() return an empty string.
	232	*/
	233	#ifndef MPACK_STRINGS
	234	#define MPACK_STRINGS 1
	235	#endif
	236
	237	/**
	238	* Set this to 1 to implement a custom @ref mpack_assert_fail() function.
	239	* See the documentation on @ref mpack_assert_fail() for details.
	240	*
	241	* Asserts are only used when @ref MPACK_DEBUG is enabled, and can be
	242	* triggered by bugs in MPack or bugs due to incorrect usage of MPack.
	243	*/
	244	#ifndef MPACK_CUSTOM_ASSERT
	245	#define MPACK_CUSTOM_ASSERT 0
	246	#endif
	247
	248	/**
	249	* @def MPACK_READ_TRACKING
	250	*
	251	* Enables compound type size tracking for readers. This ensures that the
	252	* correct number of elements or bytes are read from a compound type.
	253	*
	254	* This is enabled by default in debug builds (provided a @c malloc() is
	255	* available.)
	256	*/
	257	#if !defined(MPACK_READ_TRACKING) && \
	258	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	259	defined(MPACK_READER) && MPACK_READER && \
	260	defined(MPACK_MALLOC)
	261	#define MPACK_READ_TRACKING 1
	262	#endif
	263
	264	/**
	265	* @def MPACK_WRITE_TRACKING
	266	*
	267	* Enables compound type size tracking for writers. This ensures that the
	268	* correct number of elements or bytes are written in a compound type.
	269	*
	270	* Note that without write tracking enabled, it is possible for buggy code
	271	* to emit invalid MessagePack without flagging an error by writing the wrong
	272	* number of elements or bytes in a compound type. With tracking enabled,
	273	* MPack will catch such errors and break on the offending line of code.
	274	*
	275	* This is enabled by default in debug builds (provided a @c malloc() is
	276	* available.)
	277	*/
	278	#if !defined(MPACK_WRITE_TRACKING) && \
	279	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	280	defined(MPACK_WRITER) && MPACK_WRITER && \
	281	defined(MPACK_MALLOC)
	282	#define MPACK_WRITE_TRACKING 1
	283	#endif
	284
	285	/**
	286	* @}
	287	*/
	288
	289
	290	/**
	291	* @name Miscellaneous Options
	292	* @{
	293	*/
	294
	295	/**
	296	* Whether to optimize for size or speed.
	297	*
	298	* Optimizing for size simplifies some parsing and encoding algorithms
	299	* at the expense of speed, and saves a few kilobytes of space in the
	300	* resulting executable.
	301	*
	302	* This automatically detects -Os with GCC/Clang. Unfortunately there
	303	* doesn't seem to be a macro defined for /Os under MSVC.
	304	*/
	305	#ifndef MPACK_OPTIMIZE_FOR_SIZE
	306	#ifdef __OPTIMIZE_SIZE__
	307	#define MPACK_OPTIMIZE_FOR_SIZE 1
	308	#else
	309	#define MPACK_OPTIMIZE_FOR_SIZE 0
	310	#endif
	311	#endif
	312
	313	/**
	314	* Stack space in bytes to use when initializing a reader or writer
	315	* with a stack-allocated buffer.
	316	*/
	317	#ifndef MPACK_STACK_SIZE
	318	#define MPACK_STACK_SIZE 4096
	319	#endif
	320
	321	/**
	322	* Buffer size to use for allocated buffers (such as for a file writer.)
	323	*
	324	* Starting with a single page and growing as needed seems to
	325	* provide the best performance with minimal memory waste.
	326	* Increasing this does not improve performance even when writing
	327	* huge messages.
	328	*/
	329	#ifndef MPACK_BUFFER_SIZE
	330	#define MPACK_BUFFER_SIZE 4096
	331	#endif
	332
	333	/**
	334	* Minimum size of an allocated node page in bytes.
	335	*
	336	* The children for a given compound element must be contiguous, so
	337	* larger pages than this may be allocated as needed. (Safety checks
	338	* exist to prevent malicious data from causing too large allocations.)
	339	*
	340	* See @ref mpack_node_data_t for the size of nodes.
	341	*
	342	* Using as many nodes fit in one memory page seems to provide the
	343	* best performance, and has very little waste when parsing small
	344	* messages.
	345	*/
	346	#ifndef MPACK_NODE_PAGE_SIZE
	347	#define MPACK_NODE_PAGE_SIZE 4096
	348	#endif
	349
	350	/**
	351	* The initial depth for the node parser. When MPACK_MALLOC is available,
	352	* the node parser has no practical depth limit, and it is not recursive
	353	* so there is no risk of overflowing the call stack.
	354	*/
	355	#ifndef MPACK_NODE_INITIAL_DEPTH
	356	#define MPACK_NODE_INITIAL_DEPTH 8
	357	#endif
	358
	359	/**
	360	* The maximum depth for the node parser if @ref MPACK_MALLOC is not available.
	361	*/
	362	#ifndef MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC
	363	#define MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC 32
	364	#endif
	365
	366	/**
	367	* @}
	368	*/
	369
	370
	371	/**
	372	* @}
	373	*/
	374

+60

-0

backend/balboa-rocksdb/Makefile less more

	0
	1	CROSS_HOST?=$(shell uname -m)
	2	CROSS_PREFIX?=
	3	CCOMPILER?=gcc
	4
	5	OUT=build/
	6
	7	CFLAGS?=
	8	CFLAGS+=-pipe -Ofast -flto
	9
	10	ifdef DEBUG
	11	CFLAGS+=-ggdb
	12	else
	13	CFLAGS+=-s
	14	endif
	15
	16	ifeq ($(CCOMPILER),gcc)
	17	CFLAGS+=-fwhole-program -fmax-errors=3 -D__GCC__
	18	endif
	19
	20	ifeq ($(CCOMPILER),clang)
	21	CFLAGS+=-D__CLANG__
	22	endif
	23
	24	CFLAGS+=-std=c11 -Wall -Wextra -D_GNU_SOURCE -D__TRACE__ -DNDEBUG
	25	CFLAGS+=-I. -I../lib
	26	CFLAGS+=-DMPACK_HAS_CONFIG
	27
	28	LDFLAGS?=
	29	LDFLAGS+=-lrocksdb -pthread
	30
	31	MAKEFLAGS+=--no-print-directory
	32
	33	CC=$(CROSS_PREFIX)$(CCOMPILER)
	34
	35	hdr-balboa-rocksdb=protocol.h engine.h trace.h daemon.h mpack.h mpack-config.h
	36	hdr-balboa-rocksdb-y=$(addprefix ../lib/,$(hdr-balboa-rocksdb)) rocksdb-impl.h
	37
	38	src-balboa-rocksdb=trace.c daemon.c protocol.c engine.c mpack.c
	39	src-balboa-rocksdb-y=$(addprefix ../lib/,$(src-balboa-rocksdb))
	40	src-balboa-rocksdb-y+=rocksdb-impl.c main.c
	41
	42	target-balboa-rocksdb-y=$(OUT)$(CROSS_PREFIX)balboa-rocksdb
	43
	44	dirs-y=.
	45
	46	all: $(target-balboa-rocksdb-y)
	47
	48	$(OUT)build:
	49	@echo " mkdir"
	50	$(Q)mkdir -p $(addprefix $(OUT),$(dirs-y))
	51	$(Q)touch $@
	52
	53	$(target-balboa-rocksdb-y): $(OUT)build $(src-balboa-rocksdb-y) $(hdr-balboa-rocksdb-y) Makefile
	54	$(CC) $(CFLAGS) $(src-balboa-rocksdb-y) -o $(target-balboa-rocksdb-y) $(LDFLAGS)
	55
	56	clean:
	57	rm -f $(target-balboa-rocksdb-y)
	58	rm -f $(OUT)build
	59	rmdir $(OUT)

+121

-0

backend/balboa-rocksdb/main.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <engine.h>
	4	#include <ketopt.h>
	5	#include <rocksdb-impl.h>
	6	#include <trace.h>
	7	#include <unistd.h>
	8
	9	__attribute__((noreturn)) void version(void) {
	10	fprintf(stderr, "balboa-rocksdb v2.0.0\n");
	11	exit(1);
	12	}
	13
	14	__attribute__((noreturn)) void usage(const blb_rocksdb_config_t* c) {
	15	fprintf(
	16	stderr,
	17	"\
	18	`balboa-rocksdb` provides a pdns database backend for `balboa`\n\
	19	\n\
	20	Usage: balboa-rocksdb [options]\n\
	21	\n\
	22	-h display help\n\
	23	-D daemonize (default: off)\n\
	24	-d <path> path to rocksdb database (default: `%s`)\n\
	25	-l listen address (default: 127.0.0.1)\n\
	26	-p listen port (default: 4242)\n\
	27	-v increase verbosity; can be passed multiple times\n\
	28	-S disable signal handling\n\
	29	-R disable engine stats reporter\n\
	30	-j connection throttle limit, maximum concurrent connections (default: 64)\n\
	31	--membudget <memory-in-bytes> rocksdb membudget option (value: %" PRIu64
	32	")\n\
	33	--parallelism <number-of-threads> rocksdb parallelism option (value: %d)\n\
	34	--max_log_file_size <size> rocksdb log file size option (value: %" PRIu64
	35	")\n\
	36	--max_open_files <number> rocksdb max number of open files (value: %d)\n\
	37	--keep_log_file_num <number> rocksdb max number of log files (value: %d)\n\
	38	--database_path <path> same as `-d`\n\
	39	--version show version then exit\n\
	40	\n",
	41	c->path,
	42	c->membudget,
	43	c->parallelism,
	44	c->max_log_file_size,
	45	c->max_open_files,
	46	c->keep_log_file_num);
	47	exit(1);
	48	}
	49
	50	int main(int argc, char** argv) {
	51	int daemonize = 0;
	52	blb_rocksdb_config_t rocksdb_config = blb_rocksdb_config_init();
	53	engine_config_t engine_config = blb_engine_server_config_init();
	54	trace_config_t trace_config = {.stream = stderr,
	55	.host = "pdns",
	56	.app = "balboa-rocksdb",
	57	// leaking process number ...
	58	.procid = getpid(),
	59	.verbosity = 0};
	60	ketopt_t opt = KETOPT_INIT;
	61	static ko_longopt_t opts[] = {
	62	{"membudget", ko_required_argument, 301},
	63	{"parallelism", ko_required_argument, 302},
	64	{"max_log_file_size", ko_required_argument, 303},
	65	{"max_open_files", ko_required_argument, 304},
	66	{"keep_log_file_num", ko_required_argument, 305},
	67	{"database_path", ko_required_argument, 306},
	68	{"version", ko_no_argument, 307},
	69	{NULL, 0, 0}};
	70	int c;
	71	while((c = ketopt(&opt, argc, argv, 1, "j:d:l:p:vDSRh", opts)) >= 0) {
	72	switch(c) {
	73	case 'D': daemonize = 1; break;
	74	case 'd': rocksdb_config.path = opt.arg; break;
	75	case 'l': engine_config.host = opt.arg; break;
	76	case 'p': engine_config.port = atoi(opt.arg); break;
	77	case 'v': trace_config.verbosity += 1; break;
	78	case 'j': engine_config.conn_throttle_limit = atoi(opt.arg); break;
	79	case 'S': engine_config.enable_signal_consumer = false; break;
	80	case 'R': engine_config.enable_stats_reporter = false; break;
	81	case 'h': usage(&rocksdb_config);
	82	case 301: rocksdb_config.membudget = atoll(opt.arg); break;
	83	case 302: rocksdb_config.parallelism = atoi(opt.arg); break;
	84	case 303: rocksdb_config.max_log_file_size = atoi(opt.arg); break;
	85	case 304: rocksdb_config.max_open_files = atoi(opt.arg); break;
	86	case 305: rocksdb_config.keep_log_file_num = atoi(opt.arg); break;
	87	case 306: rocksdb_config.path = opt.arg; break;
	88	case 307: version();
	89	default: usage(&rocksdb_config);
	90	}
	91	}
	92
	93	theTrace_stream_use(&trace_config);
	94	if(daemonize
	95	&& (trace_config.stream == stderr \|\| trace_config.stream == stdout)) {
	96	theTrace_set_verbosity(0);
	97	}
	98
	99	db_t* db = blb_rocksdb_open(&rocksdb_config);
	100	if(db == NULL) {
	101	L(log_error("unable to open rocksdb at path `%s`", rocksdb_config.path));
	102	return (1);
	103	}
	104
	105	engine_config.db = db;
	106	engine_t* e = blb_engine_server_new(&engine_config);
	107	if(e == NULL) {
	108	L(log_error("unable to create engine"));
	109	blb_dbi_teardown(db);
	110	return (1);
	111	}
	112
	113	blb_engine_run(e);
	114
	115	blb_engine_teardown(e);
	116
	117	blb_dbi_teardown(db);
	118
	119	return (0);
	120	}⏎

+887

-0

backend/balboa-rocksdb/rocksdb-impl.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <stdio.h>
	4	#include <stdlib.h>
	5	#include <string.h>
	6
	7	#include <rocksdb-impl.h>
	8	#include <rocksdb/c.h>
	9
	10	#define ROCKSDB_CONN_SCRTCH_SZ (1024 * 10)
	11
	12	static void blb_rocksdb_teardown(db_t* _db);
	13	static db_t* blb_rocksdb_conn_init(conn_t* th, db_t* db);
	14	static void blb_rocksdb_conn_deinit(conn_t* th, db_t* db);
	15	static int blb_rocksdb_query(conn_t* th, const protocol_query_request_t* q);
	16	static int blb_rocksdb_input(conn_t* th, const protocol_input_request_t* i);
	17	static void blb_rocksdb_backup(conn_t* th, const protocol_backup_request_t* b);
	18	static void blb_rocksdb_dump(conn_t* th, const protocol_dump_request_t* d);
	19
	20	static const dbi_t blb_rocksdb_dbi = {.thread_init = blb_rocksdb_conn_init,
	21	.thread_deinit = blb_rocksdb_conn_deinit,
	22	.teardown = blb_rocksdb_teardown,
	23	.query = blb_rocksdb_query,
	24	.input = blb_rocksdb_input,
	25	.backup = blb_rocksdb_backup,
	26	.dump = blb_rocksdb_dump};
	27
	28	struct blb_rocksdb_t {
	29	const dbi_t* dbi;
	30	rocksdb_t* db;
	31	rocksdb_options_t* options;
	32	rocksdb_writeoptions_t* writeoptions;
	33	rocksdb_readoptions_t* readoptions;
	34	rocksdb_mergeoperator_t* mergeop;
	35	};
	36
	37	typedef struct blb_rocksdb_conn_t blb_rocksdb_conn_t;
	38	struct blb_rocksdb_conn_t {
	39	char scrtch_key[ROCKSDB_CONN_SCRTCH_SZ];
	40	char scrtch_inv[ROCKSDB_CONN_SCRTCH_SZ];
	41	};
	42
	43	rocksdb_t* blb_rocksdb_handle(db_t* db);
	44	rocksdb_readoptions_t* blb_rocksdb_readoptions(db_t* db);
	45
	46	typedef struct value_t value_t;
	47	struct value_t {
	48	uint32_t count;
	49	uint32_t first_seen;
	50	uint32_t last_seen;
	51	};
	52
	53	static inline value_t blb_rocksdb_val_init() {
	54	return ((value_t){.count = 0, .first_seen = UINT32_MAX, .last_seen = 0});
	55	}
	56
	57	#define blb_rocksdb_max(a, b) \
	58	({ \
	59	__typeof__(a) _a = (a); \
	60	__typeof__(b) _b = (b); \
	61	_a > _b ? _a : _b; \
	62	})
	63
	64	#define blb_rocksdb_min(a, b) \
	65	({ \
	66	__typeof__(a) _a = (a); \
	67	__typeof__(b) _b = (b); \
	68	_a < _b ? _a : _b; \
	69	})
	70
	71	static inline void _write_u32_le(unsigned char* p, uint32_t v) {
	72	p[0] = v >> 0;
	73	p[1] = v >> 8;
	74	p[2] = v >> 16;
	75	p[3] = v >> 24;
	76	}
	77
	78	static inline uint32_t _read_u32_le(const unsigned char* p) {
	79	return (
	80	(((uint32_t)p[0]) << 0) \| (((uint32_t)p[1]) << 8)
	81	\| (((uint32_t)p[2]) << 16) \| (((uint32_t)p[3]) << 24));
	82	}
	83
	84	static inline int blb_rocksdb_val_encode(
	85	const struct value_t* o, char* buf, size_t buflen) {
	86	size_t minlen = sizeof(uint32_t) * 3;
	87	if(buflen < minlen) { return (-1); }
	88
	89	unsigned char* p = (unsigned char*)buf;
	90	_write_u32_le(p + 0, o->count);
	91	_write_u32_le(p + 4, o->last_seen);
	92	_write_u32_le(p + 8, o->first_seen);
	93
	94	return (0);
	95	}
	96
	97	static inline blb_rocksdb_conn_t* blb_rocksdb_get_conn(conn_t* conn) {
	98	ASSERT(
	99	conn->usr_ctx != NULL && conn->usr_ctx_sz == sizeof(blb_rocksdb_conn_t));
	100	return ((blb_rocksdb_conn_t*)(conn->usr_ctx));
	101	}
	102
	103	static inline int blb_rocksdb_val_decode(
	104	value_t* o, const char* buf, size_t buflen) {
	105	size_t minlen = sizeof(uint32_t) * 3;
	106	if(buflen < minlen) { return (-1); };
	107
	108	const unsigned char* p = (const unsigned char*)buf;
	109	o->count = _read_u32_le(p + 0);
	110	o->last_seen = _read_u32_le(p + 4);
	111	o->first_seen = _read_u32_le(p + 8);
	112
	113	return (0);
	114	}
	115
	116	static inline void blb_rocksdb_val_merge(value_t* lhs, const value_t* rhs) {
	117	lhs->count += rhs->count;
	118	lhs->last_seen = blb_rocksdb_max(lhs->last_seen, rhs->last_seen);
	119	lhs->first_seen = blb_rocksdb_min(lhs->first_seen, rhs->first_seen);
	120	}
	121
	122	static char* blb_rocksdb_merge_fully(
	123	void* state,
	124	const char* key,
	125	size_t key_len,
	126	value_t* obs,
	127	const char* const* opnds,
	128	const size_t* opnds_len,
	129	int n_opnds,
	130	unsigned char* success,
	131	size_t* new_len) {
	132	(void)state;
	133	if(key_len < 5) {
	134	V(log_warn(
	135	"merge called on unknown key `%p` `%s` `%.*s` opnds `%d`",
	136	key,
	137	key,
	138	(int)key_len,
	139	key,
	140	n_opnds));
	141	}
	142	// this is an observation value
	143	size_t buf_length = sizeof(uint32_t) * 3;
	144	char* buf = malloc(buf_length);
	145	if(buf == NULL) {
	146	*success = (unsigned char)0;
	147	*new_len = 0;
	148	return (NULL);
	149	}
	150	for(int i = 0; i < n_opnds; i++) {
	151	value_t nobs = {0, 0, 0};
	152	int rc = blb_rocksdb_val_decode(&nobs, opnds[i], opnds_len[i]);
	153	if(rc != 0) {
	154	L(log_error(
	155	"blb_rocksdb_val_decode() failed (key `%.*s` opnd `%d`)",
	156	(int)key_len,
	157	key,
	158	i));
	159	continue;
	160	}
	161	blb_rocksdb_val_merge(obs, &nobs);
	162	}
	163	blb_rocksdb_val_encode(obs, buf, buf_length);
	164	*new_len = buf_length;
	165	*success = (unsigned char)1;
	166	return (buf);
	167	}
	168
	169	static char* blb_rocksdb_mergeop_full_merge(
	170	void* state,
	171	const char* key,
	172	size_t key_len,
	173	const char* existing_value,
	174	size_t existing_value_length,
	175	const char* const* opnds,
	176	const size_t* opnds_len,
	177	int n_opnds,
	178	unsigned char* success,
	179	size_t* new_len) {
	180	value_t obs = blb_rocksdb_val_init();
	181	if(key[0] == 'o' && existing_value != NULL) {
	182	int rc =
	183	blb_rocksdb_val_decode(&obs, existing_value, existing_value_length);
	184	if(rc != 0) {
	185	L(log_error("blb_rocksdb_val_decode() failed"));
	186	*success = 1;
	187	return (NULL);
	188	}
	189	}
	190	char* result = blb_rocksdb_merge_fully(
	191	state, key, key_len, &obs, opnds, opnds_len, n_opnds, success, new_len);
	192	return (result);
	193	}
	194
	195	static char* blb_rocksdb_mergeop_partial_merge(
	196	void* state,
	197	const char* key,
	198	size_t key_len,
	199	const char* const* opnds,
	200	const size_t* opnds_len,
	201	int n_opnds,
	202	unsigned char* success,
	203	size_t* new_len) {
	204	value_t obs = blb_rocksdb_val_init();
	205	char* result = blb_rocksdb_merge_fully(
	206	state, key, key_len, &obs, opnds, opnds_len, n_opnds, success, new_len);
	207	return (result);
	208	}
	209
	210	static void blb_rocksdb_mergeop_destructor(void* state) {
	211	(void)state;
	212	}
	213
	214	static const char* blb_rocksdb_mergeop_name(void* state) {
	215	(void)state;
	216	return ("observation-mergeop");
	217	}
	218
	219	static inline rocksdb_mergeoperator_t* blb_rocksdb_mergeoperator_create() {
	220	return (rocksdb_mergeoperator_create(
	221	NULL,
	222	blb_rocksdb_mergeop_destructor,
	223	blb_rocksdb_mergeop_full_merge,
	224	blb_rocksdb_mergeop_partial_merge,
	225	NULL,
	226	blb_rocksdb_mergeop_name));
	227	}
	228
	229	db_t* blb_rocksdb_conn_init(conn_t* th, db_t* db) {
	230	blb_rocksdb_conn_t* conn = blb_new(blb_rocksdb_conn_t);
	231	if(conn == NULL) { return (NULL); }
	232	th->usr_ctx = conn;
	233	th->usr_ctx_sz = sizeof(blb_rocksdb_conn_t);
	234	return (db);
	235	}
	236
	237	void blb_rocksdb_conn_deinit(conn_t* th, db_t* db) {
	238	ASSERT(db->dbi == &blb_rocksdb_dbi);
	239	ASSERT(th->usr_ctx != NULL && th->usr_ctx_sz == sizeof(blb_rocksdb_conn_t));
	240	blb_free(th->usr_ctx);
	241	th->usr_ctx = NULL;
	242	th->usr_ctx_sz = 0;
	243	}
	244
	245	void blb_rocksdb_teardown(db_t* _db) {
	246	ASSERT(_db->dbi == &blb_rocksdb_dbi);
	247	blb_rocksdb_t* db = (blb_rocksdb_t*)_db;
	248	L(log_notice("teardown"));
	249	rocksdb_mergeoperator_destroy(db->mergeop);
	250	rocksdb_writeoptions_destroy(db->writeoptions);
	251	rocksdb_readoptions_destroy(db->readoptions);
	252	// keeping this causes segfault
	253	// rocksdb_options_destroy(db->options);
	254	rocksdb_close(db->db);
	255	blb_free(db);
	256	}
	257
	258	static int blb_rocksdb_query_by_o(
	259	conn_t* th, const protocol_query_request_t* q) {
	260	ASSERT(th->db->dbi == &blb_rocksdb_dbi);
	261	blb_rocksdb_conn_t* dbc = blb_rocksdb_get_conn(th);
	262	blb_rocksdb_t* db = (blb_rocksdb_t*)th->db;
	263	size_t prefix_len = 0;
	264	if(q->qsensorid_len > 0) {
	265	prefix_len = q->qsensorid_len + q->qrrname_len + 4;
	266	(void)snprintf(
	267	dbc->scrtch_key,
	268	ROCKSDB_CONN_SCRTCH_SZ,
	269	"o\x1f%.s\x1f%.s\x1f",
	270	(int)q->qrrname_len,
	271	q->qrrname,
	272	(int)q->qsensorid_len,
	273	q->qsensorid);
	274	} else {
	275	prefix_len = q->qrrname_len + 3;
	276	(void)snprintf(
	277	dbc->scrtch_key,
	278	ROCKSDB_CONN_SCRTCH_SZ,
	279	"o\x1f%.*s\x1f",
	280	(int)q->qrrname_len,
	281	q->qrrname);
	282	}
	283
	284	X(log_debug("prefix key `%.*s`", (int)prefix_len, dbc->scrtch_key));
	285
	286	int start_ok = blb_conn_query_stream_start_response(th);
	287	if(start_ok != 0) {
	288	L(log_error("unable to start query stream response"));
	289	return (-1);
	290	}
	291
	292	rocksdb_iterator_t* it = rocksdb_create_iterator(db->db, db->readoptions);
	293	rocksdb_iter_seek(it, dbc->scrtch_key, prefix_len);
	294	size_t keys_visited = 0;
	295	size_t keys_hit = 0;
	296	for(;
	297	rocksdb_iter_valid(it) != (unsigned char)0 && keys_hit < (size_t)q->limit;
	298	rocksdb_iter_next(it)) {
	299	keys_visited += 1;
	300	size_t key_len = 0;
	301	const char* key = rocksdb_iter_key(it, &key_len);
	302	if(key == NULL) {
	303	L(log_error("impossible: unable to extract key from rocksdb iterator"));
	304	goto stream_error;
	305	}
	306
	307	enum TokIdx { RRNAME = 0, SENSORID = 1, RRTYPE = 2, RDATA = 3, FIELDS = 4 };
	308
	309	struct Tok {
	310	const char* tok;
	311	int tok_len;
	312	};
	313
	314	struct Tok toks[FIELDS];
	315	memset(toks, 0, sizeof(toks));
	316
	317	enum TokIdx j = RRNAME;
	318	size_t last = 1;
	319	for(size_t i = 2; i < key_len; i++) {
	320	if(key[i] == '\x1f') {
	321	// we fixup the RDATA and skip extra \x1f's
	322	if(j < RDATA) {
	323	toks[j].tok = &key[last + 1];
	324	toks[j].tok_len = i - last - 1;
	325	last = i;
	326	j++;
	327	}
	328	}
	329	}
	330	toks[RDATA].tok = &key[last + 1];
	331	toks[RDATA].tok_len = key_len - last - 1;
	332
	333	X(log_debug(
	334	"o %.s %.s %.s %.s",
	335	toks[RRNAME].tok_len,
	336	toks[RRNAME].tok,
	337	toks[SENSORID].tok_len,
	338	toks[SENSORID].tok,
	339	toks[RRTYPE].tok_len,
	340	toks[RRTYPE].tok,
	341	toks[RDATA].tok_len,
	342	toks[RDATA].tok));
	343
	344	size_t qrrname_len = q->qrrname_len;
	345	if(toks[RRNAME].tok_len <= 0
	346	\|\| memcmp(
	347	toks[RRNAME].tok,
	348	q->qrrname,
	349	blb_rocksdb_min((size_t)toks[RRNAME].tok_len, qrrname_len))
	350	!= 0) {
	351	break;
	352	}
	353	if((size_t)toks[RRNAME].tok_len != qrrname_len) { break; }
	354
	355	if(toks[SENSORID].tok_len == 0
	356	\|\| (q->qsensorid_len > 0
	357	&& (size_t)toks[SENSORID].tok_len != q->qsensorid_len)
	358	\|\| (q->qsensorid_len > 0
	359	&& memcmp(toks[SENSORID].tok, q->qsensorid, toks[SENSORID].tok_len)
	360	!= 0)) {
	361	continue;
	362	}
	363
	364	if(toks[RDATA].tok_len == 0
	365	\|\| (q->qrdata_len > 0 && (size_t)toks[RDATA].tok_len != q->qrdata_len)
	366	\|\| (q->qrdata_len > 0
	367	&& memcmp(toks[RDATA].tok, q->qrdata, toks[RDATA].tok_len) != 0)) {
	368	continue;
	369	}
	370
	371	if(toks[RRTYPE].tok_len == 0
	372	\|\| (q->qrrtype_len > 0 && (size_t)toks[RRTYPE].tok_len != q->qrrtype_len)
	373	\|\| (q->qrrtype_len > 0
	374	&& memcmp(toks[RRTYPE].tok, q->qrrtype, toks[RRTYPE].tok_len)
	375	!= 0)) {
	376	continue;
	377	}
	378
	379	size_t val_size = 0;
	380	value_t v;
	381	const char* val = rocksdb_iter_value(it, &val_size);
	382	int ret = blb_rocksdb_val_decode(&v, val, val_size);
	383	if(ret != 0) {
	384	L(log_error("blb_rocksdb_val_decode() failed"));
	385	continue;
	386	}
	387
	388	keys_hit += 1;
	389	protocol_entry_t __e, *e = &__e;
	390	e->sensorid = toks[SENSORID].tok;
	391	e->sensorid_len = toks[SENSORID].tok_len;
	392	e->rdata = toks[RDATA].tok;
	393	e->rdata_len = toks[RDATA].tok_len;
	394	e->rrname = toks[RRNAME].tok;
	395	e->rrname_len = toks[RRNAME].tok_len;
	396	e->rrtype = toks[RRTYPE].tok;
	397	e->rrtype_len = toks[RRTYPE].tok_len;
	398	e->count = v.count;
	399	e->first_seen = v.first_seen;
	400	e->last_seen = v.last_seen;
	401	int push_ok = blb_conn_query_stream_push_response(th, e);
	402	if(push_ok != 0) {
	403	L(log_error("unable to push query response entry"));
	404	goto stream_error;
	405	}
	406	}
	407	char* err = NULL;
	408	rocksdb_iter_get_error(it, &err);
	409	if(err != NULL) {
	410	L(log_error("iterator error `%s`", err));
	411	free(err);
	412	}
	413	rocksdb_iter_destroy(it);
	414	(void)blb_conn_query_stream_end_response(th);
	415	T(log_debug("keys_visited `%zu` keys_hit `%zu`", keys_visited, keys_hit));
	416	return (0);
	417
	418	stream_error:
	419	rocksdb_iter_destroy(it);
	420	return (-1);
	421	}
	422
	423	static int blb_rocksdb_query_by_i(
	424	conn_t* th, const protocol_query_request_t* q) {
	425	ASSERT(th->db->dbi == &blb_rocksdb_dbi);
	426	blb_rocksdb_conn_t* dbc = blb_rocksdb_get_conn(th);
	427	blb_rocksdb_t* db = (blb_rocksdb_t*)th->db;
	428	size_t prefix_len = 0;
	429	if(q->qsensorid_len > 0) {
	430	prefix_len = q->qrdata_len + q->qsensorid_len + 4;
	431	(void)snprintf(
	432	dbc->scrtch_inv,
	433	ROCKSDB_CONN_SCRTCH_SZ,
	434	"i\x1f%.s\x1f%.s\x1f",
	435	(int)q->qrdata_len,
	436	q->qrdata,
	437	(int)q->qsensorid_len,
	438	q->qsensorid);
	439	} else {
	440	prefix_len = q->qrdata_len + 3;
	441	(void)snprintf(
	442	dbc->scrtch_inv,
	443	ROCKSDB_CONN_SCRTCH_SZ,
	444	"i\x1f%.*s\x1f",
	445	(int)q->qrdata_len,
	446	q->qrdata);
	447	}
	448
	449	ASSERT(dbc->scrtch_inv[prefix_len] == '\0');
	450
	451	T(log_debug("prefix key `%.*s`", (int)prefix_len, dbc->scrtch_inv));
	452
	453	int start_ok = blb_conn_query_stream_start_response(th);
	454	if(start_ok != 0) {
	455	L(log_error("unable to start query stream response"));
	456	return (-1);
	457	}
	458
	459	rocksdb_iterator_t* it = rocksdb_create_iterator(db->db, db->readoptions);
	460	rocksdb_iter_seek(it, dbc->scrtch_inv, prefix_len);
	461	size_t keys_visited = 0;
	462	size_t keys_hit = 0;
	463	for(;
	464	rocksdb_iter_valid(it) != (unsigned char)0 && keys_hit < (size_t)q->limit;
	465	rocksdb_iter_next(it)) {
	466	keys_visited += 1;
	467	size_t key_len = 0;
	468	const char* key = rocksdb_iter_key(it, &key_len);
	469	char* err = NULL;
	470
	471	enum TokIdx { RDATA = 3, SENSORID = 2, RRNAME = 1, RRTYPE = 0, FIELDS = 4 };
	472
	473	struct Tok {
	474	const char* tok;
	475	int tok_len;
	476	};
	477
	478	struct Tok toks[FIELDS];
	479	memset(toks, 0, sizeof(toks));
	480
	481	enum TokIdx j = RRTYPE;
	482	size_t last = key_len;
	483	for(ssize_t i = key_len - 1; i > 0; i--) {
	484	if(key[i] == '\x1f') {
	485	if(j < FIELDS) {
	486	toks[j].tok = &key[i + 1];
	487	toks[j].tok_len = last - i - 1;
	488	last = i;
	489	j++;
	490	}
	491	}
	492	}
	493	toks[RDATA].tok = key + 2;
	494	toks[RDATA].tok_len = toks[RDATA].tok_len + last - 1;
	495
	496	X(log_debug("k `%zu` `%.*s`", key_len, (int)key_len, key));
	497	X(log_debug(
	498	"i `%.s` \| `%.s` `%.s` `%.s`",
	499	toks[RDATA].tok_len,
	500	toks[RDATA].tok,
	501	toks[SENSORID].tok_len,
	502	toks[SENSORID].tok,
	503	toks[RRTYPE].tok_len,
	504	toks[RRTYPE].tok,
	505	toks[RRNAME].tok_len,
	506	toks[RRNAME].tok));
	507
	508	if(j < FIELDS) {
	509	L(log_error("found invalid key `%.*s`; skipping ...", (int)key_len, key));
	510	continue;
	511	}
	512
	513	size_t qrdata_len = q->qrdata_len;
	514	if(toks[RDATA].tok_len <= 0
	515	\|\| memcmp(
	516	toks[RDATA].tok,
	517	q->qrdata,
	518	blb_rocksdb_min((size_t)toks[RDATA].tok_len, qrdata_len))
	519	!= 0) {
	520	break;
	521	}
	522	if((size_t)toks[RDATA].tok_len != qrdata_len) { break; }
	523
	524	if(toks[SENSORID].tok_len == 0
	525	\|\| (q->qsensorid_len > 0
	526	&& (size_t)toks[SENSORID].tok_len != q->qsensorid_len)
	527	\|\| (q->qsensorid_len > 0
	528	&& memcmp(toks[SENSORID].tok, q->qsensorid, toks[SENSORID].tok_len)
	529	!= 0)) {
	530	continue;
	531	}
	532
	533	if(toks[RRTYPE].tok_len == 0
	534	\|\| (q->qrrtype_len > 0 && (size_t)toks[RRTYPE].tok_len != q->qrrtype_len)
	535	\|\| (q->qrrtype_len > 0
	536	&& memcmp(toks[RRTYPE].tok, q->qrrtype, toks[RRTYPE].tok_len)
	537	!= 0)) {
	538	continue;
	539	}
	540
	541	memset(dbc->scrtch_key, '\0', ROCKSDB_CONN_SCRTCH_SZ);
	542	ssize_t fullkey_len = snprintf(
	543	dbc->scrtch_key,
	544	ROCKSDB_CONN_SCRTCH_SZ,
	545	"o\x1f%.s\x1f%.s\x1f%.s\x1f%.s",
	546	toks[RRNAME].tok_len,
	547	toks[RRNAME].tok,
	548	toks[SENSORID].tok_len,
	549	toks[SENSORID].tok,
	550	toks[RRTYPE].tok_len,
	551	toks[RRTYPE].tok,
	552	toks[RDATA].tok_len,
	553	toks[RDATA].tok);
	554
	555	if(fullkey_len <= 0 \|\| fullkey_len >= ROCKSDB_CONN_SCRTCH_SZ) {
	556	L(log_error("invalid key `%.*s`", (int)fullkey_len, dbc->scrtch_key));
	557	continue;
	558	}
	559
	560	X(log_debug("full key `%.*s`", (int)fullkey_len, dbc->scrtch_key));
	561
	562	size_t val_size = 0;
	563	char* val = rocksdb_get(
	564	db->db, db->readoptions, dbc->scrtch_key, fullkey_len, &val_size, &err);
	565	if(val == NULL \|\| err != NULL) {
	566	X(log_debug("rocksdb_get() failed with `%s`", err));
	567	free(err);
	568	continue;
	569	}
	570
	571	value_t v;
	572	int ret = blb_rocksdb_val_decode(&v, val, val_size);
	573	if(ret != 0) {
	574	L(log_error(
	575	"blb_rocksdb_val_decode() failed (key `%.*s` val_ptr `%p` val_sz "
	576	"`%zu`)",
	577	(int)fullkey_len,
	578	dbc->scrtch_key,
	579	val,
	580	val_size));
	581	free(val);
	582	continue;
	583	}
	584	free(val);
	585
	586	keys_hit += 1;
	587	protocol_entry_t __e, *e = &__e;
	588	e->sensorid = toks[SENSORID].tok;
	589	e->sensorid_len = toks[SENSORID].tok_len;
	590	e->rdata = toks[RDATA].tok;
	591	e->rdata_len = toks[RDATA].tok_len;
	592	e->rrname = toks[RRNAME].tok;
	593	e->rrname_len = toks[RRNAME].tok_len;
	594	e->rrtype = toks[RRTYPE].tok;
	595	e->rrtype_len = toks[RRTYPE].tok_len;
	596	e->count = v.count;
	597	e->first_seen = v.first_seen;
	598	e->last_seen = v.last_seen;
	599	int push_ok = blb_conn_query_stream_push_response(th, e);
	600	if(push_ok != 0) {
	601	L(log_error("unable to push query response entry"));
	602	goto stream_error;
	603	}
	604	}
	605	char* err = NULL;
	606	rocksdb_iter_get_error(it, &err);
	607	if(err != NULL) {
	608	L(log_error("iterator error `%s`", err));
	609	free(err);
	610	}
	611	rocksdb_iter_destroy(it);
	612	(void)blb_conn_query_stream_end_response(th);
	613	T(log_debug("keys_visited `%zu` keys_hit `%zu`", keys_visited, keys_hit));
	614	return (0);
	615
	616	stream_error:
	617	rocksdb_iter_destroy(it);
	618	return (-1);
	619	}
	620
	621	static int blb_rocksdb_query(conn_t* th, const protocol_query_request_t* q) {
	622	int rc = -1;
	623	if(q->qrrname_len > 0) {
	624	rc = blb_rocksdb_query_by_o(th, q);
	625	} else {
	626	rc = blb_rocksdb_query_by_i(th, q);
	627	}
	628	return (rc);
	629	}
	630
	631	static void blb_rocksdb_backup(conn_t* th, const protocol_backup_request_t* b) {
	632	ASSERT(th->db->dbi == &blb_rocksdb_dbi);
	633	blb_rocksdb_t* db = (blb_rocksdb_t*)th->db;
	634
	635	X(log_info("backup `%.*s`", (int)b->path_len, b->path));
	636
	637	if(b->path_len >= 256) {
	638	L(log_error("invalid path"));
	639	return;
	640	}
	641
	642	char path[256];
	643	snprintf(path, sizeof(path), "%.*s", (int)b->path_len, b->path);
	644
	645	char* err = NULL;
	646	rocksdb_backup_engine_t* be =
	647	rocksdb_backup_engine_open(db->options, path, &err);
	648	if(err != NULL) {
	649	L(log_error("rocksdb_backup_engine_open() failed `%s`", err));
	650	free(err);
	651	return;
	652	}
	653
	654	rocksdb_backup_engine_create_new_backup(be, db->db, &err);
	655	if(err != NULL) {
	656	L(log_error("rocksdb_backup_engine_create_new_backup() failed `%s`", err));
	657	free(err);
	658	rocksdb_backup_engine_close(be);
	659	return;
	660	}
	661	}
	662
	663	static void blb_rocksdb_dump(conn_t* th, const protocol_dump_request_t* d) {
	664	ASSERT(th->db->dbi == &blb_rocksdb_dbi);
	665	blb_rocksdb_t* db = (blb_rocksdb_t*)th->db;
	666
	667	X(log_info("dump `%.*s`", (int)d->path_len, d->path));
	668
	669	uint64_t cnt = 0;
	670	rocksdb_iterator_t* it = rocksdb_create_iterator(db->db, db->readoptions);
	671	// rocksdb_iter_seek_to_first(it);
	672	rocksdb_iter_seek(it, "o", 1);
	673	for(; rocksdb_iter_valid(it) != (unsigned char)0; rocksdb_iter_next(it)) {
	674	size_t key_len = 0;
	675	const char* key = rocksdb_iter_key(it, &key_len);
	676	if(key == NULL) {
	677	L(log_error("impossible: unable to extract key from rocksdb iterator"));
	678	break;
	679	}
	680
	681	if(key[0] == 'i') { continue; }
	682
	683	enum TokIdx { RRNAME = 0, SENSORID = 1, RRTYPE = 2, RDATA = 3, FIELDS = 4 };
	684
	685	struct Tok {
	686	const char* tok;
	687	int tok_len;
	688	};
	689
	690	struct Tok toks[FIELDS];
	691	memset(toks, 0, sizeof(toks));
	692
	693	enum TokIdx j = RRNAME;
	694	size_t last = 1;
	695	for(size_t i = 2; i < key_len; i++) {
	696	if(key[i] == '\x1f') {
	697	// we fixup the RDATA and skip extra \x1f's
	698	if(j < RDATA) {
	699	toks[j].tok = &key[last + 1];
	700	toks[j].tok_len = i - last - 1;
	701	last = i;
	702	j++;
	703	}
	704	}
	705	}
	706	toks[RDATA].tok = &key[last + 1];
	707	toks[RDATA].tok_len = key_len - last - 1;
	708
	709	X(log_debug(
	710	"o %.s %.s %.s %.s",
	711	toks[RRNAME].tok_len,
	712	toks[RRNAME].tok,
	713	toks[SENSORID].tok_len,
	714	toks[SENSORID].tok,
	715	toks[RRTYPE].tok_len,
	716	toks[RRTYPE].tok,
	717	toks[RDATA].tok_len,
	718	toks[RDATA].tok));
	719
	720	size_t val_size = 0;
	721	value_t v;
	722	const char* val = rocksdb_iter_value(it, &val_size);
	723	int ret = blb_rocksdb_val_decode(&v, val, val_size);
	724	if(ret != 0) {
	725	L(log_error("blb_rocksdb_val_decode() failed"));
	726	continue;
	727	}
	728
	729	cnt += 1;
	730	protocol_entry_t __e, *e = &__e;
	731	e->sensorid = toks[SENSORID].tok;
	732	e->sensorid_len = toks[SENSORID].tok_len;
	733	e->rdata = toks[RDATA].tok;
	734	e->rdata_len = toks[RDATA].tok_len;
	735	e->rrname = toks[RRNAME].tok;
	736	e->rrname_len = toks[RRNAME].tok_len;
	737	e->rrtype = toks[RRTYPE].tok;
	738	e->rrtype_len = toks[RRTYPE].tok_len;
	739	e->count = v.count;
	740	e->first_seen = v.first_seen;
	741	e->last_seen = v.last_seen;
	742
	743	int rc = blb_conn_dump_entry(th, e);
	744	if(rc != 0) {
	745	L(log_error("blb_conn_dump_entry() failed"));
	746	break;
	747	}
	748	}
	749
	750	char* err = NULL;
	751	rocksdb_iter_get_error(it, &err);
	752	if(err != NULL) {
	753	L(log_error("iterator error `%s`", err));
	754	free(err);
	755	}
	756	rocksdb_iter_destroy(it);
	757	L(log_notice("dumped `%" PRIu64 "` entries", cnt));
	758	}
	759
	760	static int blb_rocksdb_input(conn_t* th, const protocol_input_request_t* i) {
	761	ASSERT(th->db->dbi == &blb_rocksdb_dbi);
	762	blb_rocksdb_t* db = (blb_rocksdb_t*)th->db;
	763	blb_rocksdb_conn_t* dbc = blb_rocksdb_get_conn(th);
	764	value_t v = {.count = i->entry.count,
	765	.first_seen = i->entry.first_seen,
	766	.last_seen = i->entry.last_seen};
	767	char val[sizeof(uint32_t) * 3];
	768	size_t val_len = sizeof(val);
	769	(void)blb_rocksdb_val_encode(&v, val, val_len);
	770
	771	int key_sz = snprintf(
	772	dbc->scrtch_key,
	773	ROCKSDB_CONN_SCRTCH_SZ,
	774	"o\x1f%.s\x1f%.s\x1f%.s\x1f%.s",
	775	(int)i->entry.rrname_len,
	776	i->entry.rrname,
	777	(int)i->entry.sensorid_len,
	778	i->entry.sensorid,
	779	(int)i->entry.rrtype_len,
	780	i->entry.rrtype,
	781	(int)i->entry.rdata_len,
	782	i->entry.rdata);
	783	if(key_sz <= 0 \|\| key_sz >= ROCKSDB_CONN_SCRTCH_SZ) {
	784	L(log_error("truncated key"));
	785	return (-1);
	786	}
	787
	788	int inv_sz = snprintf(
	789	dbc->scrtch_inv,
	790	ROCKSDB_CONN_SCRTCH_SZ,
	791	"i\x1f%.s\x1f%.s\x1f%.s\x1f%.s",
	792	(int)i->entry.rdata_len,
	793	i->entry.rdata,
	794	(int)i->entry.sensorid_len,
	795	i->entry.sensorid,
	796	(int)i->entry.rrname_len,
	797	i->entry.rrname,
	798	(int)i->entry.rrtype_len,
	799	i->entry.rrtype);
	800	if(inv_sz <= 0 \|\| inv_sz >= ROCKSDB_CONN_SCRTCH_SZ) {
	801	L(log_error("truncated inverted key"));
	802	return (-1);
	803	}
	804
	805	if(inv_sz < 5 \|\| key_sz < 5) {
	806	L(log_error(
	807	"derived invalid input keys: inv_sz `%d` key_sz `%d`", inv_sz, key_sz));
	808	return (-1);
	809	}
	810
	811	char* err = NULL;
	812	rocksdb_merge(
	813	db->db, db->writeoptions, dbc->scrtch_key, key_sz, val, val_len, &err);
	814	if(err != NULL) {
	815	L(log_error("rocksdb_merge() failed: `%s`", err));
	816	free(err);
	817	return (-1);
	818	}
	819
	820	// XXX: put vs merge
	821	rocksdb_put(db->db, db->writeoptions, dbc->scrtch_inv, inv_sz, "", 0, &err);
	822	if(err != NULL) {
	823	L(log_error("rocksdb_put() failed: `%s`", err));
	824	free(err);
	825	return (-1);
	826	}
	827
	828	return (0);
	829	}
	830
	831	rocksdb_t* blb_rocksdb_handle(db_t* _db) {
	832	ASSERT(_db->dbi == &blb_rocksdb_dbi);
	833	blb_rocksdb_t* db = (blb_rocksdb_t*)_db;
	834	return (db->db);
	835	}
	836
	837	db_t* blb_rocksdb_open(const blb_rocksdb_config_t* c) {
	838	V(log_info("rocksdb database at `%s`", c->path));
	839	V(log_info(
	840	"parallelism `%d` membudget `%zu` max_log_file_size `%zu` "
	841	"keep_log_file_num `%d`",
	842	c->parallelism,
	843	c->membudget,
	844	c->max_log_file_size,
	845	c->keep_log_file_num));
	846
	847	blb_rocksdb_t* db = blb_new(blb_rocksdb_t);
	848	if(db == NULL) { return (NULL); }
	849	db->dbi = &blb_rocksdb_dbi;
	850	char* err = NULL;
	851	int level_compression[5] = {rocksdb_lz4_compression,
	852	rocksdb_lz4_compression,
	853	rocksdb_lz4_compression,
	854	rocksdb_lz4_compression,
	855	rocksdb_lz4_compression};
	856
	857	db->mergeop = blb_rocksdb_mergeoperator_create();
	858	db->options = rocksdb_options_create();
	859	db->writeoptions = rocksdb_writeoptions_create();
	860	db->readoptions = rocksdb_readoptions_create();
	861
	862	rocksdb_options_increase_parallelism(db->options, c->parallelism);
	863	rocksdb_options_optimize_level_style_compaction(db->options, c->membudget);
	864	rocksdb_options_set_create_if_missing(db->options, 1);
	865	rocksdb_options_set_max_log_file_size(db->options, c->max_log_file_size);
	866	rocksdb_options_set_keep_log_file_num(db->options, c->keep_log_file_num);
	867	rocksdb_options_set_max_open_files(db->options, c->max_open_files);
	868	rocksdb_options_set_merge_operator(db->options, db->mergeop);
	869	rocksdb_options_set_compression_per_level(db->options, level_compression, 5);
	870
	871	db->db = rocksdb_open(db->options, c->path, &err);
	872	if(err != NULL) {
	873	L(log_error("rocksdb_open() failed: `%s`", err));
	874	rocksdb_options_destroy(db->options);
	875	rocksdb_mergeoperator_destroy(db->mergeop);
	876	rocksdb_writeoptions_destroy(db->writeoptions);
	877	rocksdb_readoptions_destroy(db->readoptions);
	878	free(err);
	879	blb_free(db);
	880	return (NULL);
	881	}
	882
	883	V(log_debug("rocksdb at %p", db));
	884
	885	return ((db_t*)db);
	886	}

+36

-0

backend/balboa-rocksdb/rocksdb-impl.h less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#ifndef __ROCKSDB_H
	4	#define __ROCKSDB_H
	5
	6	#include <stdbool.h>
	7	#include <stdint.h>
	8	#include <stdlib.h>
	9	#include <time.h>
	10
	11	#include <engine.h>
	12
	13	typedef struct blb_rocksdb_t blb_rocksdb_t;
	14	typedef struct blb_rocksdb_config_t blb_rocksdb_config_t;
	15	struct blb_rocksdb_config_t {
	16	size_t membudget;
	17	int parallelism;
	18	size_t max_log_file_size;
	19	int max_open_files;
	20	int keep_log_file_num;
	21	const char* path;
	22	};
	23
	24	static inline blb_rocksdb_config_t blb_rocksdb_config_init() {
	25	return ((blb_rocksdb_config_t){.membudget = 128 * 1024 * 1024,
	26	.parallelism = 8,
	27	.max_log_file_size = 10 * 1024 * 1024,
	28	.max_open_files = 300,
	29	.keep_log_file_num = 2,
	30	.path = "/tmp/balboa-rocksdb"});
	31	}
	32
	33	db_t* blb_rocksdb_open(const blb_rocksdb_config_t* config);
	34
	35	#endif

+138

-0

backend/go/backend.go less more

	0	// balboa-backends
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	package backend
	4
	5	import (
	6	"net"
	7	"os"
	8	"os/signal"
	9	"sync"
	10	"syscall"
	11	"time"
	12
	13	"github.com/DCSO/balboa/db"
	14	obs "github.com/DCSO/balboa/observation"
	15
	16	log "github.com/sirupsen/logrus"
	17	)
	18
	19	type Handler interface {
	20	HandleObservations(*obs.InputObservation)
	21	HandleQuery(*db.QueryRequest, net.Conn)
	22	HandleDump(*db.DumpRequest, net.Conn)
	23	HandleBackup(*db.BackupRequest)
	24	}
	25
	26	var (
	27	DecodeTimeout = 10 * time.Second
	28	QueryResultTimeout = 10 * time.Second
	29	AcceptTimeout = 10 * time.Second
	30	)
	31
	32	func handle(conn net.Conn, h Handler, stopCh chan bool, wg *sync.WaitGroup) {
	33	dec := db.MakeDecoder(conn)
	34	defer dec.Release()
	35	defer conn.Close()
	36	defer wg.Done()
	37	for {
	38	select {
	39	case <-stopCh:
	40	log.Debugf("stop signal received")
	41	return
	42	default:
	43	msg, err := dec.ExpectTypedMessage()
	44	if err != nil {
	45	log.Warnf("unable to decode typed message `%v`", err)
	46	return
	47	}
	48	switch msg.Type {
	49	case db.TypeInputRequest:
	50	log.Debugf("got input message")
	51	inner, err_inner := dec.ExpectInputRequestFromBytes(msg.EncodedMessage)
	52	if err_inner != nil {
	53	log.Warnf("unable to decode inner message: input request")
	54	return
	55	}
	56	h.HandleObservations(inner)
	57	case db.TypeQueryRequest:
	58	log.Debugf("got query message")
	59	inner, err_inner := dec.ExpectQueryRequestFromBytes(msg.EncodedMessage)
	60	if err_inner != nil {
	61	log.Warnf("unable to decode inner message: query request")
	62	return
	63	}
	64	h.HandleQuery(inner, conn)
	65	case db.TypeBackupRequest:
	66	log.Debugf("got backup request")
	67	inner, err_inner := dec.ExpectBackupRequestFromBytes(msg.EncodedMessage)
	68	if err_inner != nil {
	69	log.Warnf("unable to decode inner message: backup request")
	70	return
	71	}
	72	h.HandleBackup(inner)
	73	case db.TypeDumpRequest:
	74	log.Debugf("got dump request")
	75	inner, err_inner := dec.ExpectDumpRequestFromBytes(msg.EncodedMessage)
	76	if err_inner != nil {
	77	log.Warnf("unable to decode inner message: dump request")
	78	return
	79	}
	80	h.HandleDump(inner, conn)
	81	default:
	82	log.Warnf("invalid or unsupported message type `%v`", msg.Type)
	83	return
	84	}
	85	}
	86	}
	87	}
	88
	89	func loop(host string, h Handler, stopCh chan bool, wg *sync.WaitGroup) {
	90	log.Infof("start listening on host=%s", host)
	91	ln, err := net.Listen("tcp", host)
	92	if err != nil {
	93	log.Warnf("unable to create listening socket: %s", err)
	94	return
	95	}
	96	defer ln.Close()
	97	defer wg.Done()
	98	for {
	99	select {
	100	case <-stopCh:
	101	return
	102	default:
	103	ln.(*net.TCPListener).SetDeadline(time.Now().Add(AcceptTimeout))
	104	conn, err := ln.Accept()
	105	if err != nil {
	106	if operr, ok := err.(*net.OpError); ok && operr.Timeout() {
	107	log.Debugf("accepting client connection timeout")
	108	continue
	109	}
	110	log.Warnf("unable to accept new client connection `%s`", err)
	111	continue
	112	}
	113	log.Debugf("handling new connection")
	114	wg.Add(1)
	115	go handle(conn, h, stopCh, wg)
	116	}
	117	}
	118	}
	119
	120	func Serve(host string, h Handler) {
	121	sigCh := make(chan os.Signal, 1)
	122	done := make(chan bool, 1)
	123	signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
	124	go func() {
	125	sig := <-sigCh
	126	log.Warnf("received '%v' signal, shutting down ...", sig)
	127	close(done)
	128	}()
	129
	130	var wg sync.WaitGroup
	131
	132	wg.Add(1)
	133
	134	go loop(host, h, done, &wg)
	135
	136	wg.Wait()
	137	}

+22

-0

backend/lib/LICENSE-mpack less more

	0	The MIT License (MIT)
	1
	2	Copyright (c) 2015-2018 Nicholas Fraser
	3
	4	Permission is hereby granted, free of charge, to any person obtaining a copy
	5	of this software and associated documentation files (the "Software"), to deal
	6	in the Software without restriction, including without limitation the rights
	7	to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	8	copies of the Software, and to permit persons to whom the Software is
	9	furnished to do so, subject to the following conditions:
	10
	11	The above copyright notice and this permission notice shall be included in all
	12	copies or substantial portions of the Software.
	13
	14	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	15	IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	16	FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	17	AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	18	LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	19	OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
	20	SOFTWARE.
	21

+45

-0

backend/lib/alloc.h less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#ifndef __ALLOC_H
	4	#define __ALLOC_H
	5
	6	#include <inttypes.h>
	7	#include <stdlib.h>
	8	#include <trace.h>
	9
	10	static inline void* blb_realloc_impl(
	11	const char* name, void* p, size_t new_size) {
	12	void* pp = realloc(p, new_size);
	13	X(log_debug("(%s) realloc `%p` `%p` `%zu`", name, p, pp, new_size));
	14	return (pp);
	15	}
	16
	17	#define blb_new(ty) \
	18	({ \
	19	size_t p_sz = sizeof(ty); \
	20	void* p = malloc(p_sz); \
	21	X(log_debug("new `%s` `%zu` `%p`", #ty, p_sz, p)); \
	22	p; \
	23	})
	24	#define blb_malloc(sz) \
	25	({ \
	26	size_t p_sz = (sz); \
	27	void* p = malloc(p_sz); \
	28	X(log_debug("alloc `%zu` `%p`", p_sz, p)); \
	29	p; \
	30	})
	31	#define blb_realloc(p, sz) \
	32	({ \
	33	size_t p_sz = (sz); \
	34	void* pp = realloc((p), p_sz); \
	35	X(log_debug("realloc `%p` `%zu` `%p`", pp, p_sz, p)); \
	36	pp; \
	37	})
	38	#define blb_free(p) \
	39	do { \
	40	X(log_debug("free `%p`", (p))); \
	41	free(p); \
	42	} while(0)
	43
	44	#endif⏎

+332

-0

backend/lib/bs.h less more

	0
	1	#ifndef __BS_H
	2	#define __BS_H
	3
	4	#include <ctype.h>
	5	#include <inttypes.h>
	6	#include <stddef.h>
	7	#include <stdlib.h>
	8	#include <string.h>
	9
	10	typedef struct bytestring_source_t bytestring_source_t;
	11	struct bytestring_source_t {
	12	const unsigned char* p;
	13	const size_t available;
	14	size_t index;
	15	};
	16
	17	typedef struct bytestring_sink_t bytestring_sink_t;
	18	struct bytestring_sink_t {
	19	unsigned char* p;
	20	size_t available;
	21	size_t index;
	22	};
	23
	24	static inline uint64_t _read_w64_le( const uint8_t* p ) {
	25	return (
	26	( ( ( uint64_t )p[0] ) << 0 ) \| ( ( ( uint64_t )p[1] ) << 8 )
	27	\| ( ( ( uint64_t )p[2] ) << 16 ) \| ( ( ( uint64_t )p[3] ) << 24 )
	28	\| ( ( ( uint64_t )p[4] ) << 32 ) \| ( ( ( uint64_t )p[5] ) << 40 )
	29	\| ( ( ( uint64_t )p[6] ) << 48 ) \| ( ( ( uint64_t )p[7] ) << 56 ) );
	30	}
	31
	32	static inline uint64_t _read_w64_be( uint8_t* p ) {
	33	return ( __builtin_bswap64( _read_w64_le( p ) ) );
	34	}
	35
	36	static inline uint32_t _read_w32_le( const uint8_t* p ) {
	37	return (
	38	( ( ( uint32_t )p[0] ) << 0 ) \| ( ( ( uint32_t )p[1] ) << 8 )
	39	\| ( ( ( uint32_t )p[2] ) << 16 ) \| ( ( ( uint32_t )p[3] ) << 24 ) );
	40	}
	41
	42	static inline uint32_t _read_w32_be( const uint8_t* p ) {
	43	return ( __builtin_bswap32( _read_w32_le( p ) ) );
	44	}
	45
	46	static inline uint16_t _read_w16_le( const uint8_t* p ) {
	47	return ( ( ( ( uint16_t )p[0] ) << 0 ) \| ( ( ( uint16_t )p[1] ) << 8 ) );
	48	}
	49
	50	static inline uint16_t _read_w16_be( const uint8_t* p ) {
	51	return ( __builtin_bswap16( _read_w16_le( p ) ) );
	52	}
	53
	54	static inline void _write_w64_le( uint8_t* p, uint64_t v ) {
	55	p[0] = v >> 0;
	56	p[1] = v >> 8;
	57	p[2] = v >> 16;
	58	p[3] = v >> 24;
	59	p[4] = v >> 32;
	60	p[5] = v >> 40;
	61	p[6] = v >> 48;
	62	p[7] = v >> 56;
	63	}
	64
	65	static inline void _write_w32_le( uint8_t* p, uint32_t v ) {
	66	p[0] = v >> 0;
	67	p[1] = v >> 8;
	68	p[2] = v >> 16;
	69	p[3] = v >> 24;
	70	}
	71
	72	static inline void _write_w16_le( uint8_t* p, uint16_t v ) {
	73	p[0] = v >> 0;
	74	p[1] = v >> 8;
	75	}
	76
	77	static inline uint32_t bs_read_w16_le( bytestring_source_t* bs, int* ok ) {
	78	if( bs->index + 2 > bs->available ) {
	79	if( ok != NULL ) { *ok = -1; }
	80	return ( 0 );
	81	}
	82	uint64_t x = _read_w16_le( bs->p + bs->index );
	83	bs->index += 2;
	84	return ( x );
	85	}
	86
	87	static inline uint32_t bs_read_w32_le( bytestring_source_t* bs, int* ok ) {
	88	if( bs->index + 4 > bs->available ) {
	89	if( ok != NULL ) { *ok = -1; }
	90	return ( 0 );
	91	}
	92	uint64_t x = _read_w32_le( bs->p + bs->index );
	93	bs->index += 4;
	94	return ( x );
	95	}
	96
	97	static inline uint64_t bs_read_w64_le( bytestring_source_t* bs, int* ok ) {
	98	if( bs->index + 8 > bs->available ) {
	99	if( ok != NULL ) { *ok = -1; }
	100	return ( 0 );
	101	}
	102	uint64_t x = _read_w64_le( bs->p + bs->index );
	103	bs->index += 8;
	104	return ( x );
	105	}
	106
	107	static inline uint64_t bs_read_w64_be( bytestring_source_t* bs, int* ok ) {
	108	return ( __builtin_bswap32( bs_read_w64_le( bs, ok ) ) );
	109	}
	110
	111	static inline int bs_write_w16_le( bytestring_sink_t* bs, uint16_t x ) {
	112	if( bs->index + 2 > bs->available ) { return ( -1 ); }
	113	_write_w16_le( bs->p + bs->index, x );
	114	bs->index += 2;
	115	return ( x );
	116	}
	117
	118	static inline int bs_write_w32_le( bytestring_sink_t* bs, uint32_t x ) {
	119	if( bs->index + 4 > bs->available ) { return ( -1 ); }
	120	_write_w32_le( bs->p + bs->index, x );
	121	bs->index += 4;
	122	return ( x );
	123	}
	124
	125	static inline int bs_write_w64_le( bytestring_sink_t* bs, uint64_t x ) {
	126	if( bs->index + 8 > bs->available ) { return ( -1 ); }
	127	_write_w64_le( bs->p + bs->index, x );
	128	bs->index += 8;
	129	return ( x );
	130	}
	131
	132	static inline int bs_append(
	133	bytestring_sink_t* bs, const uint8_t* p, size_t n ) {
	134	if( n > bs->available ) { return ( -1 ); }
	135	if( bs->index + n < bs->index ) { return ( -1 ); }
	136	if( bs->index + n > bs->available ) { return ( -1 ); }
	137	memmove( bs->p + bs->index, p, n );
	138	bs->index += n;
	139	return ( 0 );
	140	}
	141
	142	static inline int bs_cat( bytestring_sink_t* bs, const char* p, size_t n ) {
	143	return ( bs_append( bs, ( const uint8_t* )p, n ) );
	144	}
	145
	146	static inline int bs_append1( bytestring_sink_t* bs, uint8_t x ) {
	147	if( bs->available < 1 ) { return ( -1 ); }
	148	if( bs->index + 1 < bs->index ) { return ( -1 ); }
	149	if( bs->index + 1 > bs->available ) { return ( -1 ); }
	150	bs->p[bs->index] = x;
	151	bs->index += 1;
	152	return ( 0 );
	153	}
	154
	155	static inline int bs_slurp( bytestring_source_t* bs, uint8_t* p, size_t n ) {
	156	if( n > ( bs->available - bs->index ) ) { return ( -1 ); }
	157	memmove( p, bs->p + bs->index, n );
	158	bs->index += n;
	159	return ( 0 );
	160	}
	161
	162	static inline bytestring_sink_t bs_sink_slice0(
	163	const bytestring_sink_t* bs, size_t n ) {
	164	if( bs->available < n ) { return ( ( bytestring_sink_t ){0} ); }
	165	if( bs->index + n < bs->index ) { return ( ( bytestring_sink_t ){0} ); }
	166	if( bs->index + n > bs->available ) { return ( ( bytestring_sink_t ){0} ); }
	167	return ( ( bytestring_sink_t ){
	168	.p = bs->p + bs->index + n, .index = 0, .available = bs->available - n} );
	169	}
	170
	171	#define bs_source( data, n ) \
	172	( bytestring_source_t ) { \
	173	.p = ( data ), .index = 0, .available = ( n ) \
	174	}
	175	#define bs_sink( data, n ) \
	176	( bytestring_sink_t ) { \
	177	.p = ( data ), .index = 0, .available = ( n ) \
	178	}
	179
	180	static inline int bs_sink_escape( bytestring_sink_t* sink, uint8_t c ) {
	181	typedef struct __escape_t __escape_t;
	182	struct __escape_t {
	183	const char* r;
	184	unsigned int l;
	185	};
	186	static __escape_t escapes[256] = {
	187	[0x00] = {.r = "\\u0000", .l = 6}, [0x01] = {.r = "\\u0001", .l = 6},
	188	[0x02] = {.r = "\\u0002", .l = 6}, [0x03] = {.r = "\\u0003", .l = 6},
	189	[0x04] = {.r = "\\u0004", .l = 6}, [0x05] = {.r = "\\u0005", .l = 6},
	190	[0x06] = {.r = "\\u0006", .l = 6}, [0x07] = {.r = "\\u0007", .l = 6},
	191	[0x08] = {.r = "\\u0008", .l = 6}, [0x09] = {.r = "\\u0009", .l = 6},
	192	[0x0a] = {.r = "\\u000a", .l = 6}, [0x0b] = {.r = "\\u000b", .l = 6},
	193	[0x0c] = {.r = "\\u000c", .l = 6}, [0x0d] = {.r = "\\u000d", .l = 6},
	194	[0x0e] = {.r = "\\u000e", .l = 6}, [0x0f] = {.r = "\\u000f", .l = 6},
	195	[0x10] = {.r = "\\u0010", .l = 6}, [0x11] = {.r = "\\u0011", .l = 6},
	196	[0x12] = {.r = "\\u0012", .l = 6}, [0x13] = {.r = "\\u0013", .l = 6},
	197	[0x14] = {.r = "\\u0014", .l = 6}, [0x15] = {.r = "\\u0015", .l = 6},
	198	[0x16] = {.r = "\\u0016", .l = 6}, [0x17] = {.r = "\\u0017", .l = 6},
	199	[0x18] = {.r = "\\u0018", .l = 6}, [0x19] = {.r = "\\u0019", .l = 6},
	200	[0x1a] = {.r = "\\u001a", .l = 6}, [0x1b] = {.r = "\\u001b", .l = 6},
	201	[0x1c] = {.r = "\\u001c", .l = 6}, [0x1d] = {.r = "\\u001d", .l = 6},
	202	[0x1e] = {.r = "\\u001e", .l = 6}, [0x1f] = {.r = "\\u001f", .l = 6},
	203	[0x20] = {.r = " ", .l = 1}, [0x21] = {.r = "!", .l = 1},
	204	[0x22] = {.r = "\\\"", .l = 2}, [0x23] = {.r = "#", .l = 1},
	205	[0x24] = {.r = "$", .l = 1}, [0x25] = {.r = "%", .l = 1},
	206	[0x26] = {.r = "&", .l = 1}, [0x27] = {.r = "'", .l = 1},
	207	[0x28] = {.r = "(", .l = 1}, [0x29] = {.r = ")", .l = 1},
	208	[0x2a] = {.r = "*", .l = 1}, [0x2b] = {.r = "+", .l = 1},
	209	[0x2c] = {.r = ",", .l = 1}, [0x2d] = {.r = "-", .l = 1},
	210	[0x2e] = {.r = ".", .l = 1}, [0x2f] = {.r = "/", .l = 1},
	211	[0x30] = {.r = "0", .l = 1}, [0x31] = {.r = "1", .l = 1},
	212	[0x32] = {.r = "2", .l = 1}, [0x33] = {.r = "3", .l = 1},
	213	[0x34] = {.r = "4", .l = 1}, [0x35] = {.r = "5", .l = 1},
	214	[0x36] = {.r = "6", .l = 1}, [0x37] = {.r = "7", .l = 1},
	215	[0x38] = {.r = "8", .l = 1}, [0x39] = {.r = "9", .l = 1},
	216	[0x3a] = {.r = ":", .l = 1}, [0x3b] = {.r = ";", .l = 1},
	217	[0x3c] = {.r = "<", .l = 1}, [0x3d] = {.r = "=", .l = 1},
	218	[0x3e] = {.r = ">", .l = 1}, [0x3f] = {.r = "?", .l = 1},
	219	[0x40] = {.r = "@", .l = 1}, [0x41] = {.r = "A", .l = 1},
	220	[0x42] = {.r = "B", .l = 1}, [0x43] = {.r = "C", .l = 1},
	221	[0x44] = {.r = "D", .l = 1}, [0x45] = {.r = "E", .l = 1},
	222	[0x46] = {.r = "F", .l = 1}, [0x47] = {.r = "G", .l = 1},
	223	[0x48] = {.r = "H", .l = 1}, [0x49] = {.r = "I", .l = 1},
	224	[0x4a] = {.r = "J", .l = 1}, [0x4b] = {.r = "K", .l = 1},
	225	[0x4c] = {.r = "L", .l = 1}, [0x4d] = {.r = "M", .l = 1},
	226	[0x4e] = {.r = "N", .l = 1}, [0x4f] = {.r = "O", .l = 1},
	227	[0x50] = {.r = "P", .l = 1}, [0x51] = {.r = "Q", .l = 1},
	228	[0x52] = {.r = "R", .l = 1}, [0x53] = {.r = "S", .l = 1},
	229	[0x54] = {.r = "T", .l = 1}, [0x55] = {.r = "U", .l = 1},
	230	[0x56] = {.r = "V", .l = 1}, [0x57] = {.r = "W", .l = 1},
	231	[0x58] = {.r = "X", .l = 1}, [0x59] = {.r = "Y", .l = 1},
	232	[0x5a] = {.r = "Z", .l = 1}, [0x5b] = {.r = "[", .l = 1},
	233	[0x5c] = {.r = "\\\\", .l = 2}, [0x5d] = {.r = "]", .l = 1},
	234	[0x5e] = {.r = "^", .l = 1}, [0x5f] = {.r = "_", .l = 1},
	235	[0x60] = {.r = "`", .l = 1}, [0x61] = {.r = "a", .l = 1},
	236	[0x62] = {.r = "b", .l = 1}, [0x63] = {.r = "c", .l = 1},
	237	[0x64] = {.r = "d", .l = 1}, [0x65] = {.r = "e", .l = 1},
	238	[0x66] = {.r = "f", .l = 1}, [0x67] = {.r = "g", .l = 1},
	239	[0x68] = {.r = "h", .l = 1}, [0x69] = {.r = "i", .l = 1},
	240	[0x6a] = {.r = "j", .l = 1}, [0x6b] = {.r = "k", .l = 1},
	241	[0x6c] = {.r = "l", .l = 1}, [0x6d] = {.r = "m", .l = 1},
	242	[0x6e] = {.r = "n", .l = 1}, [0x6f] = {.r = "o", .l = 1},
	243	[0x70] = {.r = "p", .l = 1}, [0x71] = {.r = "q", .l = 1},
	244	[0x72] = {.r = "r", .l = 1}, [0x73] = {.r = "s", .l = 1},
	245	[0x74] = {.r = "t", .l = 1}, [0x75] = {.r = "u", .l = 1},
	246	[0x76] = {.r = "v", .l = 1}, [0x77] = {.r = "w", .l = 1},
	247	[0x78] = {.r = "x", .l = 1}, [0x79] = {.r = "y", .l = 1},
	248	[0x7a] = {.r = "z", .l = 1}, [0x7b] = {.r = "{", .l = 1},
	249	[0x7c] = {.r = "\|", .l = 1}, [0x7d] = {.r = "}", .l = 1},
	250	[0x7e] = {.r = "~", .l = 1}, [0x7f] = {.r = "\\u007f", .l = 6},
	251	[0x80] = {.r = "\\u0080", .l = 6}, [0x81] = {.r = "\\u0081", .l = 6},
	252	[0x82] = {.r = "\\u0082", .l = 6}, [0x83] = {.r = "\\u0083", .l = 6},
	253	[0x84] = {.r = "\\u0084", .l = 6}, [0x85] = {.r = "\\u0085", .l = 6},
	254	[0x86] = {.r = "\\u0086", .l = 6}, [0x87] = {.r = "\\u0087", .l = 6},
	255	[0x88] = {.r = "\\u0088", .l = 6}, [0x89] = {.r = "\\u0089", .l = 6},
	256	[0x8a] = {.r = "\\u008a", .l = 6}, [0x8b] = {.r = "\\u008b", .l = 6},
	257	[0x8c] = {.r = "\\u008c", .l = 6}, [0x8d] = {.r = "\\u008d", .l = 6},
	258	[0x8e] = {.r = "\\u008e", .l = 6}, [0x8f] = {.r = "\\u008f", .l = 6},
	259	[0x90] = {.r = "\\u0090", .l = 6}, [0x91] = {.r = "\\u0091", .l = 6},
	260	[0x92] = {.r = "\\u0092", .l = 6}, [0x93] = {.r = "\\u0093", .l = 6},
	261	[0x94] = {.r = "\\u0094", .l = 6}, [0x95] = {.r = "\\u0095", .l = 6},
	262	[0x96] = {.r = "\\u0096", .l = 6}, [0x97] = {.r = "\\u0097", .l = 6},
	263	[0x98] = {.r = "\\u0098", .l = 6}, [0x99] = {.r = "\\u0099", .l = 6},
	264	[0x9a] = {.r = "\\u009a", .l = 6}, [0x9b] = {.r = "\\u009b", .l = 6},
	265	[0x9c] = {.r = "\\u009c", .l = 6}, [0x9d] = {.r = "\\u009d", .l = 6},
	266	[0x9e] = {.r = "\\u009e", .l = 6}, [0x9f] = {.r = "\\u009f", .l = 6},
	267	[0xa0] = {.r = "\\u00a0", .l = 6}, [0xa1] = {.r = "\\u00a1", .l = 6},
	268	[0xa2] = {.r = "\\u00a2", .l = 6}, [0xa3] = {.r = "\\u00a3", .l = 6},
	269	[0xa4] = {.r = "\\u00a4", .l = 6}, [0xa5] = {.r = "\\u00a5", .l = 6},
	270	[0xa6] = {.r = "\\u00a6", .l = 6}, [0xa7] = {.r = "\\u00a7", .l = 6},
	271	[0xa8] = {.r = "\\u00a8", .l = 6}, [0xa9] = {.r = "\\u00a9", .l = 6},
	272	[0xaa] = {.r = "\\u00aa", .l = 6}, [0xab] = {.r = "\\u00ab", .l = 6},
	273	[0xac] = {.r = "\\u00ac", .l = 6}, [0xad] = {.r = "\\u00ad", .l = 6},
	274	[0xae] = {.r = "\\u00ae", .l = 6}, [0xaf] = {.r = "\\u00af", .l = 6},
	275	[0xb0] = {.r = "\\u00b0", .l = 6}, [0xb1] = {.r = "\\u00b1", .l = 6},
	276	[0xb2] = {.r = "\\u00b2", .l = 6}, [0xb3] = {.r = "\\u00b3", .l = 6},
	277	[0xb4] = {.r = "\\u00b4", .l = 6}, [0xb5] = {.r = "\\u00b5", .l = 6},
	278	[0xb6] = {.r = "\\u00b6", .l = 6}, [0xb7] = {.r = "\\u00b7", .l = 6},
	279	[0xb8] = {.r = "\\u00b8", .l = 6}, [0xb9] = {.r = "\\u00b9", .l = 6},
	280	[0xba] = {.r = "\\u00ba", .l = 6}, [0xbb] = {.r = "\\u00bb", .l = 6},
	281	[0xbc] = {.r = "\\u00bc", .l = 6}, [0xbd] = {.r = "\\u00bd", .l = 6},
	282	[0xbe] = {.r = "\\u00be", .l = 6}, [0xbf] = {.r = "\\u00bf", .l = 6},
	283	[0xc0] = {.r = "\\u00c0", .l = 6}, [0xc1] = {.r = "\\u00c1", .l = 6},
	284	[0xc2] = {.r = "\\u00c2", .l = 6}, [0xc3] = {.r = "\\u00c3", .l = 6},
	285	[0xc4] = {.r = "\\u00c4", .l = 6}, [0xc5] = {.r = "\\u00c5", .l = 6},
	286	[0xc6] = {.r = "\\u00c6", .l = 6}, [0xc7] = {.r = "\\u00c7", .l = 6},
	287	[0xc8] = {.r = "\\u00c8", .l = 6}, [0xc9] = {.r = "\\u00c9", .l = 6},
	288	[0xca] = {.r = "\\u00ca", .l = 6}, [0xcb] = {.r = "\\u00cb", .l = 6},
	289	[0xcc] = {.r = "\\u00cc", .l = 6}, [0xcd] = {.r = "\\u00cd", .l = 6},
	290	[0xce] = {.r = "\\u00ce", .l = 6}, [0xcf] = {.r = "\\u00cf", .l = 6},
	291	[0xd0] = {.r = "\\u00d0", .l = 6}, [0xd1] = {.r = "\\u00d1", .l = 6},
	292	[0xd2] = {.r = "\\u00d2", .l = 6}, [0xd3] = {.r = "\\u00d3", .l = 6},
	293	[0xd4] = {.r = "\\u00d4", .l = 6}, [0xd5] = {.r = "\\u00d5", .l = 6},
	294	[0xd6] = {.r = "\\u00d6", .l = 6}, [0xd7] = {.r = "\\u00d7", .l = 6},
	295	[0xd8] = {.r = "\\u00d8", .l = 6}, [0xd9] = {.r = "\\u00d9", .l = 6},
	296	[0xda] = {.r = "\\u00da", .l = 6}, [0xdb] = {.r = "\\u00db", .l = 6},
	297	[0xdc] = {.r = "\\u00dc", .l = 6}, [0xdd] = {.r = "\\u00dd", .l = 6},
	298	[0xde] = {.r = "\\u00de", .l = 6}, [0xdf] = {.r = "\\u00df", .l = 6},
	299	[0xe0] = {.r = "\\u00e0", .l = 6}, [0xe1] = {.r = "\\u00e1", .l = 6},
	300	[0xe2] = {.r = "\\u00e2", .l = 6}, [0xe3] = {.r = "\\u00e3", .l = 6},
	301	[0xe4] = {.r = "\\u00e4", .l = 6}, [0xe5] = {.r = "\\u00e5", .l = 6},
	302	[0xe6] = {.r = "\\u00e6", .l = 6}, [0xe7] = {.r = "\\u00e7", .l = 6},
	303	[0xe8] = {.r = "\\u00e8", .l = 6}, [0xe9] = {.r = "\\u00e9", .l = 6},
	304	[0xea] = {.r = "\\u00ea", .l = 6}, [0xeb] = {.r = "\\u00eb", .l = 6},
	305	[0xec] = {.r = "\\u00ec", .l = 6}, [0xed] = {.r = "\\u00ed", .l = 6},
	306	[0xee] = {.r = "\\u00ee", .l = 6}, [0xef] = {.r = "\\u00ef", .l = 6},
	307	[0xf0] = {.r = "\\u00f0", .l = 6}, [0xf1] = {.r = "\\u00f1", .l = 6},
	308	[0xf2] = {.r = "\\u00f2", .l = 6}, [0xf3] = {.r = "\\u00f3", .l = 6},
	309	[0xf4] = {.r = "\\u00f4", .l = 6}, [0xf5] = {.r = "\\u00f5", .l = 6},
	310	[0xf6] = {.r = "\\u00f6", .l = 6}, [0xf7] = {.r = "\\u00f7", .l = 6},
	311	[0xf8] = {.r = "\\u00f8", .l = 6}, [0xf9] = {.r = "\\u00f9", .l = 6},
	312	[0xfa] = {.r = "\\u00fa", .l = 6}, [0xfb] = {.r = "\\u00fb", .l = 6},
	313	[0xfc] = {.r = "\\u00fc", .l = 6}, [0xfd] = {.r = "\\u00fd", .l = 6},
	314	[0xfe] = {.r = "\\u00fe", .l = 6}, [0xff] = {.r = "\\u00ff", .l = 6}};
	315	const __escape_t* entry = &escapes[c];
	316	return ( bs_append( sink, ( const uint8_t* )entry->r, entry->l ) );
	317	}
	318
	319	static inline int bs_append_escape(
	320	bytestring_sink_t* bs, const uint8_t* p, size_t n ) {
	321	int64_t ok = 0;
	322	for( size_t i = 0; i < n; i++ ) { ok += bs_sink_escape( bs, p[i] ); }
	323	return ( ok < 0 ? -1 : 0 );
	324	}
	325
	326	static inline int bs_cat_escape(
	327	bytestring_sink_t* bs, const char* p, size_t n ) {
	328	return ( bs_append_escape( bs, ( const uint8_t* )p, n ) );
	329	}
	330
	331	#endif⏎

+40

-0

backend/lib/daemon.c less more

	0
	1	#include <assert.h>
	2	#include <errno.h>
	3	#include <fcntl.h>
	4	#include <signal.h>
	5	#include <stdio.h>
	6	#include <stdlib.h>
	7	#include <string.h>
	8	#include <sys/types.h>
	9	#include <syslog.h>
	10	#include <unistd.h>
	11
	12	#include <daemon.h>
	13
	14	int daemonize() {
	15	pid_t pid1 = fork();
	16	if(pid1 < 0) {
	17	return (-1);
	18	} else if(pid1 != 0) {
	19	_exit(0);
	20	}
	21	pid_t sid = setsid();
	22	if(sid < 0) { return (-1); }
	23	pid_t pid2 = fork();
	24	if(pid2 < 0) {
	25	return (-1);
	26	} else if(pid2 != 0) {
	27	_exit(0);
	28	}
	29
	30	for(int n = 0; n < 1024; n++) { close(n); }
	31
	32	int fd = open("/dev/null", O_RDWR);
	33	if(fd < 0) { return (0); }
	34	dup2(fd, 0);
	35	dup2(fd, 1);
	36	dup2(fd, 2);
	37
	38	return (0);
	39	}⏎

-0

backend/lib/daemon.h less more

	0
	1	#ifndef __DAEMON_H
	2	#define __DAEMON_H
	3
	4	#include <stdlib.h>
	5
	6	int daemonize(void);
	7
	8	#endif⏎

+601

-0

backend/lib/engine.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <arpa/inet.h>
	4	#include <errno.h>
	5	#include <inttypes.h>
	6	#include <limits.h>
	7	#include <netdb.h>
	8	#include <netinet/in.h>
	9	#include <signal.h>
	10	#include <stdio.h>
	11	#include <stdlib.h>
	12	#include <string.h>
	13	#include <sys/socket.h>
	14	#include <sys/types.h>
	15	#include <unistd.h>
	16
	17	#include <engine.h>
	18	#include <protocol.h>
	19	#include <trace.h>
	20
	21	#define ENGINE_MPACK_TREE_MEMCAP (1024 * 100)
	22	#define ENGINE_MPACK_TREE_NODES_LIMIT (1024)
	23	#define ENGINE_POLL_READ_TIMEOUT (60)
	24	#define ENGINE_POLL_WRITE_TIMEOUT (30)
	25
	26	static atomic_int blb_engine_stop = ATOMIC_VAR_INIT(0);
	27	static atomic_int blb_conn_cnt = ATOMIC_VAR_INIT(0);
	28
	29	void blb_engine_request_stop() {
	30	atomic_fetch_add(&blb_engine_stop, 1);
	31	}
	32
	33	static int blb_engine_poll_stop() {
	34	return (atomic_load(&blb_engine_stop));
	35	}
	36
	37	static void blb_thread_cnt_incr() {
	38	(void)atomic_fetch_add(&blb_conn_cnt, 1);
	39	}
	40
	41	static void blb_thread_cnt_decr() {
	42	(void)atomic_fetch_sub(&blb_conn_cnt, 1);
	43	}
	44
	45	static int blb_thread_cnt_get() {
	46	return (atomic_load(&blb_conn_cnt));
	47	}
	48
	49	static inline int blb_engine_poll_write(int fd, int seconds) {
	50	timeout_retry:
	51	if(blb_engine_poll_stop() > 0) {
	52	L(log_notice("engine stop detected"));
	53	return (-1);
	54	}
	55	fd_set fds;
	56	struct timeval to;
	57	FD_ZERO(&fds);
	58	FD_SET(fd, &fds);
	59	to.tv_sec = seconds;
	60	to.tv_usec = 0;
	61	int rc = select(fd + 1, NULL, &fds, NULL, &to);
	62	if(rc == 0) {
	63	X(log_debug("select() timeout => retry polling"));
	64	goto timeout_retry;
	65	} else if(rc < 0) {
	66	X(log_debug("select() failed `%s`", strerror(errno)));
	67	return (-1);
	68	}
	69	return (0);
	70	}
	71
	72	static inline int blb_engine_poll_read(int fd, int seconds) {
	73	timeout_retry:
	74	if(blb_engine_poll_stop() > 0) {
	75	L(log_notice("engine stop detected"));
	76	return (-1);
	77	}
	78	fd_set fds;
	79	struct timeval to;
	80	FD_ZERO(&fds);
	81	FD_SET(fd, &fds);
	82	to.tv_sec = seconds;
	83	to.tv_usec = 0;
	84	int rc = select(fd + 1, &fds, NULL, NULL, &to);
	85	if(rc == 0) {
	86	X(log_debug("select() timeout => retry polling"));
	87	goto timeout_retry;
	88	} else if(rc < 0) {
	89	X(log_debug("select() failed `%s`", strerror(errno)));
	90	return (-1);
	91	}
	92	return (0);
	93	}
	94
	95	int blb_conn_write_all(conn_t* th, char* _p, size_t _p_sz) {
	96	int wr_ok = blb_engine_poll_write(th->fd, ENGINE_POLL_WRITE_TIMEOUT);
	97	if(wr_ok != 0) {
	98	L(log_error("blb_engine_poll_write() failed"));
	99	return (-1);
	100	}
	101	char* p = _p;
	102	ssize_t r = _p_sz;
	103	while(r > 0) {
	104	ssize_t rc = write(th->fd, p, r);
	105	if(rc < 0) {
	106	L(log_error("write() failed error `%s`", strerror(errno)));
	107	return (-1);
	108	} else if(rc == 0 && errno == EINTR) {
	109	continue;
	110	}
	111	r -= rc;
	112	p += rc;
	113	}
	114	blb_engine_stats_add(th->engine, ENGINE_STATS_BYTES_SEND, _p_sz);
	115	return (0);
	116	}
	117
	118	int blb_conn_query_stream_start_response(conn_t* th) {
	119	if(blb_engine_poll_stop() > 0) {
	120	L(log_notice("thread <%04lx> engine stop detected", th->thread));
	121	return (-1);
	122	}
	123
	124	ssize_t used = blb_protocol_encode_stream_start_response(
	125	th->scrtch, ENGINE_CONN_SCRTCH_SZ);
	126	if(used <= 0) {
	127	L(log_error("blb_protocol_encode_stream_start_response() failed"));
	128	return (-1);
	129	}
	130
	131	return (blb_conn_write_all(th, th->scrtch, used));
	132	}
	133
	134	int blb_conn_dump_entry(conn_t* th, const protocol_entry_t* entry) {
	135	T(log_debug("dump stream push entry"));
	136	T(blb_protocol_log_entry(entry));
	137
	138	if(blb_engine_poll_stop() > 0) {
	139	L(log_notice("thread <%04lx> engine stop detected", th->thread));
	140	return (-1);
	141	}
	142
	143	ssize_t used =
	144	blb_protocol_encode_dump_entry(entry, th->scrtch, ENGINE_CONN_SCRTCH_SZ);
	145	if(used <= 0) {
	146	L(log_error("blb_protocol_encode_dump_entry() failed"));
	147	return (-1);
	148	}
	149
	150	return (blb_conn_write_all(th, th->scrtch, used));
	151	}
	152
	153	int blb_conn_query_stream_push_response(
	154	conn_t* th, const protocol_entry_t* entry) {
	155	T(log_debug("query stream push entry"));
	156	T(blb_protocol_log_entry(entry));
	157
	158	if(blb_engine_poll_stop() > 0) {
	159	L(log_notice("thread <%04lx> engine stop detected", th->thread));
	160	return (-1);
	161	}
	162
	163	ssize_t used = blb_protocol_encode_stream_entry(
	164	entry, th->scrtch, ENGINE_CONN_SCRTCH_SZ);
	165	if(used <= 0) {
	166	L(log_error("blb_protocol_encode_stream_entry() failed"));
	167	return (-1);
	168	}
	169
	170	return (blb_conn_write_all(th, th->scrtch, used));
	171	}
	172
	173	int blb_conn_query_stream_end_response(conn_t* th) {
	174	if(blb_engine_poll_stop() > 0) {
	175	L(log_error("thread <%04lx> engine stop detected", th->thread));
	176	return (-1);
	177	}
	178
	179	ssize_t used = blb_protocol_encode_stream_end_response(
	180	th->scrtch, ENGINE_CONN_SCRTCH_SZ);
	181	if(used <= 0) {
	182	L(log_error("blb_protocol_encode_stream_end_response() failed"));
	183	return (-1);
	184	}
	185
	186	X(log_debug(
	187	"blb_protocol_encode_stream_end_response() returned `%zd`", used));
	188
	189	return (blb_conn_write_all(th, th->scrtch, used));
	190	}
	191
	192	static conn_t* blb_engine_conn_new(engine_t* e, int fd) {
	193	conn_t* th = blb_new(conn_t);
	194	if(th == NULL) { return (NULL); }
	195	th->usr_ctx = NULL;
	196	th->usr_ctx_sz = 0;
	197	th->db = NULL;
	198	if(e->db != NULL) {
	199	db_t* db = blb_dbi_conn_init(th, e->db);
	200	if(db == NULL) {
	201	blb_free(th);
	202	return (NULL);
	203	}
	204	th->db = db;
	205	}
	206	th->engine = e;
	207	th->fd = fd;
	208	return (th);
	209	}
	210
	211	void blb_engine_conn_teardown(conn_t* th) {
	212	if(th->db != NULL) { blb_dbi_conn_deinit(th, th->db); }
	213	close(th->fd);
	214	blb_free(th);
	215	}
	216
	217	static ssize_t blb_conn_read_stream_cb(void* usr, char* p, size_t p_sz) {
	218	conn_t* th = usr;
	219	int fd_ok = blb_engine_poll_read(th->fd, ENGINE_POLL_READ_TIMEOUT);
	220	if(fd_ok != 0) {
	221	L(log_error("engine poll read failed"));
	222	return (0);
	223	}
	224	ssize_t rc = read(th->fd, p, p_sz);
	225	if(rc < 0) {
	226	blb_engine_stats_bump(th->engine, ENGINE_STATS_ERRORS);
	227	L(log_error("read() failed `%s`", strerror(errno)));
	228	return (-1);
	229	} else if(rc == 0) {
	230	X(log_debug("read() eof"));
	231	return (0);
	232	}
	233	blb_engine_stats_add(th->engine, ENGINE_STATS_BYTES_RECV, rc);
	234	return (rc);
	235	}
	236
	237	static inline int blb_engine_conn_consume_backup(
	238	conn_t* th, const protocol_backup_request_t* backup) {
	239	blb_dbi_backup(th, backup);
	240	return (0);
	241	}
	242
	243	static inline int blb_engine_conn_consume_dump(
	244	conn_t* th, const protocol_dump_request_t* dump) {
	245	blb_dbi_dump(th, dump);
	246	return (0);
	247	}
	248
	249	static inline int blb_engine_conn_consume_query(
	250	conn_t* th, const protocol_query_request_t* query) {
	251	int query_ok = blb_dbi_query(th, query);
	252	if(query_ok != 0) {
	253	L(log_error("blb_dbi_query() failed"));
	254	return (-1);
	255	}
	256	return (0);
	257	}
	258
	259	static inline int blb_engine_conn_consume_input(
	260	conn_t* th, const protocol_input_request_t* input) {
	261	T(blb_protocol_log_entry(&input->entry));
	262	int input_ok = blb_dbi_input(th, input);
	263	if(input_ok != 0) {
	264	L(log_error("blb_dbi_input() failed"));
	265	return (-1);
	266	}
	267	return (0);
	268	}
	269
	270	static inline int blb_engine_conn_consume(conn_t* th, protocol_message_t* msg) {
	271	switch(msg->ty) {
	272	case PROTOCOL_INPUT_REQUEST:
	273	blb_engine_stats_bump(th->engine, ENGINE_STATS_INPUTS);
	274	return (blb_engine_conn_consume_input(th, &msg->u.input));
	275	case PROTOCOL_BACKUP_REQUEST:
	276	blb_engine_stats_bump(th->engine, ENGINE_STATS_BACKUPS);
	277	return (blb_engine_conn_consume_backup(th, &msg->u.backup));
	278	case PROTOCOL_DUMP_REQUEST:
	279	blb_engine_stats_bump(th->engine, ENGINE_STATS_DUMPS);
	280	return (blb_engine_conn_consume_dump(th, &msg->u.dump));
	281	case PROTOCOL_QUERY_REQUEST:
	282	blb_engine_stats_bump(th->engine, ENGINE_STATS_QUERIES);
	283	return (blb_engine_conn_consume_query(th, &msg->u.query));
	284	default:
	285	blb_engine_stats_bump(th->engine, ENGINE_STATS_ERRORS);
	286	L(log_debug("invalid message type `%d`", msg->ty));
	287	return (-1);
	288	}
	289	}
	290
	291	protocol_stream_t* blb_engine_stream_new(conn_t* c) {
	292	protocol_stream_t* stream = blb_protocol_stream_new(
	293	c,
	294	blb_conn_read_stream_cb,
	295	ENGINE_MPACK_TREE_MEMCAP,
	296	ENGINE_MPACK_TREE_NODES_LIMIT);
	297	return (stream);
	298	}
	299
	300	static void* blb_engine_conn_fn(void* usr) {
	301	ASSERT(usr != NULL);
	302	conn_t* th = usr;
	303	blb_thread_cnt_incr();
	304	T(log_info("new thread is <%04lx>", th->thread));
	305
	306	protocol_stream_t* stream = blb_engine_stream_new(th);
	307	if(stream == NULL) {
	308	L(log_error("unalbe to create protocol decode stream"));
	309	goto thread_exit;
	310	}
	311
	312	protocol_message_t msg;
	313	while(1) {
	314	if(blb_engine_poll_stop() > 0) {
	315	L(log_notice("thread <%04lx> engine stop detected", th->thread));
	316	goto thread_exit;
	317	}
	318	int rc = blb_protocol_stream_decode(stream, &msg);
	319	if(rc != 0) {
	320	if(rc == -1) {
	321	X(log_debug("blb_protocol_stream_decode() eof"));
	322	} else {
	323	L(log_error("blb_protocol_stream_decode() failed"));
	324	}
	325	goto thread_exit;
	326	}
	327	int th_rc = blb_engine_conn_consume(th, &msg);
	328	if(th_rc != 0) { goto thread_exit; }
	329	if(msg.ty == PROTOCOL_DUMP_REQUEST \|\| msg.ty == PROTOCOL_BACKUP_REQUEST) {
	330	V(log_notice("closing client connection after dump or backup request"));
	331	goto thread_exit;
	332	}
	333	}
	334
	335	thread_exit:
	336	T(log_info("thread <%04lx> is shutting down", th->thread));
	337	if(stream != NULL) { blb_protocol_stream_teardown(stream); }
	338	blb_thread_cnt_decr();
	339	blb_engine_conn_teardown(th);
	340	return (NULL);
	341	}
	342
	343	engine_t* blb_engine_server_new(const engine_config_t* config) {
	344	ASSERT(config->db != NULL);
	345	ASSERT(config->is_server == true);
	346
	347	struct sockaddr_in __ipv4, *ipv4 = &__ipv4;
	348	int rc = inet_pton(AF_INET, config->host, &ipv4->sin_addr);
	349	ASSERT(rc >= 0);
	350	if(rc != 1) {
	351	errno = EINVAL;
	352	return (NULL);
	353	}
	354	ipv4->sin_family = AF_INET;
	355	ipv4->sin_port = htons((uint16_t)config->port);
	356	int fd = socket(ipv4->sin_family, SOCK_STREAM, 0);
	357	if(fd < 0) {
	358	L(log_error("socket() failed with `%s`", strerror(errno)));
	359	return (NULL);
	360	}
	361	int reuse = 1;
	362	(void)setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
	363	int bind_rc = bind(fd, ipv4, sizeof(struct sockaddr_in));
	364	if(bind_rc < 0) {
	365	L(log_error("bind() failed with `%s`", strerror(errno)));
	366	close(fd);
	367	return (NULL);
	368	}
	369	int listen_rc = listen(fd, SOMAXCONN);
	370	if(listen_rc < 0) {
	371	L(log_error("listen() failed with `%s`", strerror(errno)));
	372	close(fd);
	373	return (NULL);
	374	}
	375
	376	engine_t* e = blb_new(engine_t);
	377	if(e == NULL) {
	378	close(fd);
	379	return (NULL);
	380	}
	381
	382	e->conn_throttle_limit = config->conn_throttle_limit;
	383	e->enable_signal_consumer = config->enable_signal_consumer;
	384	e->enable_stats_reporter = config->enable_stats_reporter;
	385	e->db = config->db;
	386	e->listen_fd = fd;
	387	e->stats.interval = 10;
	388	for(int i = 0; i < ENGINE_STATS_N; i++) {
	389	atomic_store(&e->stats.counters[i], 0);
	390	}
	391
	392	V(log_info(
	393	"listening on host `%s` port `%d` fd `%d`",
	394	config->host,
	395	config->port,
	396	fd));
	397
	398	return (e);
	399	}
	400
	401	static int blb_engine_client_connect(const char* host, int port) {
	402	struct sockaddr_in addr;
	403	int addr_ok = inet_pton(AF_INET, host, &addr.sin_addr);
	404	if(addr_ok != 1) { return (-1); }
	405	addr.sin_family = AF_INET;
	406	addr.sin_port = htons((uint16_t)port);
	407	int fd = socket(addr.sin_family, SOCK_STREAM, 0);
	408	if(fd < 0) { return (-1); }
	409	int rc = connect(fd, &addr, sizeof(struct sockaddr_in));
	410	if(rc < 0) {
	411	L(log_error("connect() failed with `%s`", strerror(errno)));
	412	close(fd);
	413	return (-1);
	414	}
	415	return (fd);
	416	}
	417
	418	conn_t* blb_engine_client_new(const engine_config_t* config) {
	419	ASSERT(config->db == NULL);
	420	ASSERT(config->is_server == false);
	421	int fd = blb_engine_client_connect(config->host, config->port);
	422	if(fd < 0) { return (NULL); }
	423
	424	engine_t* e = blb_new(engine_t);
	425	if(e == NULL) {
	426	close(fd);
	427	return (NULL);
	428	}
	429	e->db = NULL;
	430
	431	conn_t* c = blb_engine_conn_new(e, fd);
	432	if(c == NULL) {
	433	L(log_error("blb_engine_conn_new() failed"));
	434	blb_free(e);
	435	return (NULL);
	436	}
	437
	438	return (c);
	439	}
	440
	441	static void* blb_engine_signal_consume(void* usr) {
	442	(void)usr;
	443	sigset_t s;
	444	sigemptyset(&s);
	445	sigaddset(&s, SIGQUIT);
	446	sigaddset(&s, SIGUSR1);
	447	sigaddset(&s, SIGUSR2);
	448	sigaddset(&s, SIGINT);
	449	sigaddset(&s, SIGPIPE);
	450	sigaddset(&s, SIGTERM);
	451	V(log_info("signal consumer thread started"));
	452	while(1) {
	453	int sig = 0;
	454	int rc = sigwait(&s, &sig);
	455	V(log_debug("sigwait() returned `%d` (signal `%d`)", rc, sig));
	456	if(rc != 0) { continue; }
	457	L(log_notice("received signal `%d`", sig));
	458	switch(sig) {
	459	case SIGINT:
	460	case SIGTERM:
	461	case SIGQUIT:
	462	L(log_warn("requesting engine stop due to received signal"));
	463	blb_engine_request_stop();
	464	break;
	465	default: L(log_warn("ignoring signal")); break;
	466	}
	467	}
	468	return (NULL);
	469	}
	470
	471	static inline unsigned long long blb_stats_slurp(
	472	engine_stats_t* s, enum engine_stats_counter_t c) {
	473	if(c < 0 \|\| c >= ENGINE_STATS_N) { return (ULLONG_MAX); }
	474	unsigned long long x = atomic_load(&s->counters[c]);
	475	atomic_store(&s->counters[c], 0);
	476	return (x);
	477	}
	478
	479	static void* blb_engine_stats_report(void* usr) {
	480	V(log_info("engine stats reporter thread started"));
	481	engine_t* e = usr;
	482	blb_thread_cnt_incr();
	483	pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
	484	pthread_cond_t c = PTHREAD_COND_INITIALIZER;
	485	clock_gettime(CLOCK_REALTIME, &e->stats.last);
	486	while(1) {
	487	if(blb_engine_poll_stop() > 0) {
	488	L(log_notice("engine stats stop detected"));
	489	blb_thread_cnt_decr();
	490	return (NULL);
	491	}
	492	(void)pthread_mutex_lock(&m);
	493	struct timespec ts;
	494	clock_gettime(CLOCK_REALTIME, &ts);
	495	ts.tv_sec += e->stats.interval;
	496	(void)pthread_cond_timedwait(&c, &m, &ts);
	497	clock_gettime(CLOCK_REALTIME, &ts);
	498	long delta_t = ts.tv_sec - e->stats.last.tv_sec;
	499	L(log_notice(
	500	"delta_t `%ld` q `%llu` i `%llu` e `%llu` s `%llu` r `%llu` c `%llu`",
	501	delta_t,
	502	blb_stats_slurp(&e->stats, ENGINE_STATS_QUERIES),
	503	blb_stats_slurp(&e->stats, ENGINE_STATS_INPUTS),
	504	blb_stats_slurp(&e->stats, ENGINE_STATS_ERRORS),
	505	blb_stats_slurp(&e->stats, ENGINE_STATS_BYTES_SEND),
	506	blb_stats_slurp(&e->stats, ENGINE_STATS_BYTES_RECV),
	507	blb_stats_slurp(&e->stats, ENGINE_STATS_CONNECTIONS)));
	508	e->stats.last = ts;
	509	(void)pthread_mutex_unlock(&m);
	510	}
	511	blb_thread_cnt_decr();
	512	return (NULL);
	513	}
	514
	515	void blb_engine_spawn_signal_consumer(engine_t* e) {
	516	sigset_t s;
	517	sigemptyset(&s);
	518	sigaddset(&s, SIGQUIT);
	519	sigaddset(&s, SIGUSR1);
	520	sigaddset(&s, SIGUSR2);
	521	sigaddset(&s, SIGINT);
	522	sigaddset(&s, SIGPIPE);
	523	sigaddset(&s, SIGTERM);
	524	int rc = pthread_sigmask(SIG_BLOCK, &s, NULL);
	525	if(rc != 0) { L(log_error("pthread_sigmask() failed `%d`", rc)); }
	526	pthread_create(&e->signal_consumer, NULL, blb_engine_signal_consume, e);
	527	}
	528
	529	void blb_engine_spawn_stats_reporter(engine_t* e) {
	530	pthread_create(&e->stats_reporter, NULL, blb_engine_stats_report, e);
	531	}
	532
	533	void blb_engine_run(engine_t* e) {
	534	struct sockaddr_in __addr, *addr = &__addr;
	535	socklen_t addrlen = sizeof(struct sockaddr_in);
	536
	537	if(e->enable_signal_consumer) { blb_engine_spawn_signal_consumer(e); }
	538
	539	if(e->enable_stats_reporter) { blb_engine_spawn_stats_reporter(e); }
	540
	541	pthread_attr_t __attr;
	542	pthread_attr_init(&__attr);
	543	pthread_attr_setdetachstate(&__attr, PTHREAD_CREATE_DETACHED);
	544
	545	fd_set fds;
	546	struct timeval to;
	547	while(1) {
	548	timeout_retry:
	549	if(blb_engine_poll_stop() > 0) {
	550	L(log_notice("engine stop detected"));
	551	goto teardown;
	552	}
	553	if(blb_thread_cnt_get() >= e->conn_throttle_limit) {
	554	blb_engine_sleep(1);
	555	L(log_warn("thread throttle reached"));
	556	goto timeout_retry;
	557	}
	558	FD_ZERO(&fds);
	559	FD_SET(e->listen_fd, &fds);
	560	to.tv_sec = 5;
	561	to.tv_usec = 0;
	562	int rc = select(e->listen_fd + 1, &fds, NULL, NULL, &to);
	563	if(rc == 0) {
	564	X(log_debug("select() timeout"));
	565	goto timeout_retry;
	566	} else if(rc < 0) {
	567	L(log_error("select() failed `%s`", strerror(errno)));
	568	goto teardown;
	569	}
	570	socket_t fd = accept(e->listen_fd, (struct sockaddr*)addr, &addrlen);
	571	if(fd < 0) {
	572	L(log_error("accept() failed: `%s`", strerror(errno)));
	573	blb_engine_request_stop();
	574	goto teardown;
	575	}
	576	conn_t* th = blb_engine_conn_new(e, fd);
	577	if(th == NULL) {
	578	L(log_error("blb_engine_conn_new() failed"));
	579	blb_engine_request_stop();
	580	goto teardown;
	581	}
	582	blb_engine_stats_bump(th->engine, ENGINE_STATS_CONNECTIONS);
	583	(void)pthread_create(&th->thread, &__attr, blb_engine_conn_fn, (void*)th);
	584	}
	585
	586	teardown:
	587
	588	pthread_attr_destroy(&__attr);
	589
	590	while(blb_thread_cnt_get() > 0) {
	591	L(log_warn("waiting for `%d` thread(s) to finish", blb_thread_cnt_get()));
	592	blb_engine_sleep(2);
	593	}
	594
	595	L(log_notice("all threads finished"));
	596	}
	597
	598	void blb_engine_teardown(engine_t* e) {
	599	blb_free(e);
	600	}

+181

-0

backend/lib/engine.h less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#ifndef __ENGINE_H
	4	#define __ENGINE_H
	5
	6	#include <alloc.h>
	7	#include <inttypes.h>
	8	#include <protocol.h>
	9	#include <pthread.h>
	10	#include <stdatomic.h>
	11	#include <time.h>
	12	#include <trace.h>
	13
	14	#define ENGINE_CONN_SCRTCH_SZ (1024 * 10)
	15	#define ENGINE_CONN_SCRTCH_BUFFERS (10)
	16
	17	typedef int socket_t;
	18
	19	typedef struct dbi_t dbi_t;
	20	typedef struct db_t db_t;
	21	typedef struct engine_t engine_t;
	22	typedef struct conn_t conn_t;
	23	typedef struct engine_stats_t engine_stats_t;
	24
	25	#define QUERY_TYPE_DEFAULT 0
	26
	27	struct dbi_t {
	28	db_t* (thread_init)(conn_t, db_t*);
	29	void (thread_deinit)(conn_t, db_t*);
	30	void (teardown)(db_t db);
	31	int (query)(conn_t th, const protocol_query_request_t* query);
	32	int (input)(conn_t th, const protocol_input_request_t* input);
	33	void (backup)(conn_t th, const protocol_backup_request_t* backup);
	34	void (dump)(conn_t th, const protocol_dump_request_t* dump);
	35	};
	36
	37	struct db_t {
	38	const dbi_t* dbi;
	39	};
	40
	41	enum engine_stats_counter_t {
	42	ENGINE_STATS_QUERIES = 0,
	43	ENGINE_STATS_INPUTS = 1,
	44	ENGINE_STATS_BACKUPS = 2,
	45	ENGINE_STATS_DUMPS = 3,
	46	ENGINE_STATS_BYTES_RECV = 4,
	47	ENGINE_STATS_BYTES_SEND = 5,
	48	ENGINE_STATS_CONNECTIONS = 6,
	49	ENGINE_STATS_ERRORS = 7,
	50	ENGINE_STATS_N = 8
	51	};
	52
	53	struct engine_stats_t {
	54	unsigned long interval;
	55	struct timespec last;
	56	atomic_ullong counters[ENGINE_STATS_N];
	57	};
	58
	59	struct engine_t {
	60	engine_stats_t stats;
	61	int conn_throttle_limit;
	62	db_t* db;
	63	socket_t listen_fd;
	64	bool enable_stats_reporter;
	65	bool enable_signal_consumer;
	66	pthread_t stats_reporter;
	67	pthread_t signal_consumer;
	68	};
	69
	70	struct conn_t {
	71	pthread_t thread;
	72	engine_t* engine;
	73	db_t* db;
	74	void* usr_ctx;
	75	size_t usr_ctx_sz;
	76	socket_t fd;
	77	char scrtch[ENGINE_CONN_SCRTCH_SZ];
	78	};
	79
	80	static inline void blb_engine_sleep(long seconds) {
	81	pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
	82	pthread_cond_t c = PTHREAD_COND_INITIALIZER;
	83	(void)pthread_mutex_lock(&m);
	84	struct timespec ts;
	85	clock_gettime(CLOCK_REALTIME, &ts);
	86	ts.tv_sec += seconds;
	87	(void)pthread_cond_timedwait(&c, &m, &ts);
	88	(void)pthread_mutex_unlock(&m);
	89	}
	90
	91	static inline db_t* blb_dbi_conn_init(conn_t* th, db_t* db) {
	92	return (db->dbi->thread_init(th, db));
	93	}
	94
	95	static inline void blb_dbi_conn_deinit(conn_t* th, db_t* db) {
	96	db->dbi->thread_deinit(th, db);
	97	}
	98
	99	static inline void blb_dbi_teardown(db_t* db) {
	100	db->dbi->teardown(db);
	101	}
	102
	103	static inline int blb_dbi_query(conn_t* th, const protocol_query_request_t* q) {
	104	return (th->db->dbi->query(th, q));
	105	}
	106
	107	static inline int blb_dbi_input(conn_t* th, const protocol_input_request_t* i) {
	108	return (th->db->dbi->input(th, i));
	109	}
	110
	111	static inline void blb_dbi_backup(
	112	conn_t* th, const protocol_backup_request_t* b) {
	113	th->db->dbi->backup(th, b);
	114	}
	115
	116	static inline void blb_dbi_dump(conn_t* th, const protocol_dump_request_t* d) {
	117	th->db->dbi->dump(th, d);
	118	}
	119
	120	static inline void blb_engine_stats_bump(
	121	engine_t* engine, enum engine_stats_counter_t counter) {
	122	if(counter < 0 \|\| counter >= ENGINE_STATS_N) { return; }
	123	atomic_fetch_add(&engine->stats.counters[counter], 1);
	124	}
	125
	126	static inline void blb_engine_stats_add(
	127	engine_t* engine,
	128	enum engine_stats_counter_t counter,
	129	unsigned long long x) {
	130	if(counter < 0 \|\| counter >= ENGINE_STATS_N) { return; }
	131	atomic_fetch_add(&engine->stats.counters[counter], x);
	132	}
	133
	134	typedef struct engine_config_t engine_config_t;
	135	struct engine_config_t {
	136	int conn_throttle_limit;
	137	bool is_server;
	138	bool enable_signal_consumer;
	139	bool enable_stats_reporter;
	140	db_t* db;
	141	const char* host;
	142	int port;
	143	};
	144
	145	static inline engine_config_t blb_engine_server_config_init() {
	146	return ((engine_config_t){.db = NULL,
	147	.conn_throttle_limit = 64,
	148	.is_server = true,
	149	.enable_stats_reporter = true,
	150	.enable_signal_consumer = true,
	151	.host = "127.0.0.1",
	152	.port = 4242});
	153	}
	154
	155	static inline engine_config_t blb_engine_client_config_init() {
	156	return ((engine_config_t){.db = NULL,
	157	.conn_throttle_limit = 64,
	158	.is_server = false,
	159	.enable_stats_reporter = true,
	160	.enable_signal_consumer = true,
	161	.host = "127.0.0.1",
	162	.port = 4242});
	163	}
	164
	165	void blb_engine_signals_init(void);
	166	engine_t* blb_engine_server_new(const engine_config_t* config);
	167	conn_t* blb_engine_client_new(const engine_config_t* config);
	168	void blb_engine_teardown(engine_t* e);
	169	void blb_engine_run(engine_t* e);
	170	void blb_engine_request_stop(void);
	171	protocol_stream_t* blb_engine_stream_new(conn_t* c);
	172	int blb_conn_write_all(conn_t* th, char* _p, size_t _p_sz);
	173	void blb_engine_conn_teardown(conn_t* th);
	174
	175	int blb_conn_query_stream_start_response(conn_t* th);
	176	int blb_conn_query_stream_push_response(conn_t, const protocol_entry_t entry);
	177	int blb_conn_query_stream_end_response(conn_t* th);
	178	int blb_conn_dump_entry(conn_t* th, const protocol_entry_t* entry);
	179
	180	#endif⏎

+120

-0

backend/lib/ketopt.h less more

	0	#ifndef KETOPT_H
	1	#define KETOPT_H
	2
	3	#include <string.h> /* for strchr() and strncmp() */
	4
	5	#define ko_no_argument 0
	6	#define ko_required_argument 1
	7	#define ko_optional_argument 2
	8
	9	typedef struct {
	10	int ind; /* equivalent to optind */
	11	int opt; /* equivalent to optopt */
	12	char arg; / equivalent to optarg */
	13	int longidx; /* index of a long option; or -1 if short */
	14	/* private variables not intended for external uses */
	15	int i, pos, n_args;
	16	} ketopt_t;
	17
	18	typedef struct {
	19	char *name;
	20	int has_arg;
	21	int val;
	22	} ko_longopt_t;
	23
	24	static ketopt_t KETOPT_INIT = { 1, 0, 0, -1, 1, 0, 0 };
	25
	26	static void ketopt_permute(char argv[], int j, int n) / move argv[j] over n elements to the left */
	27	{
	28	int k;
	29	char *p = argv[j];
	30	for (k = 0; k < n; ++k)
	31	argv[j - k] = argv[j - k - 1];
	32	argv[j - k] = p;
	33	}
	34
	35	/**
	36	* Parse command-line options and arguments
	37	*
	38	* This fuction has a similar interface to GNU's getopt_long(). Each call
	39	* parses one option and returns the option name. s->arg points to the option
	40	* argument if present. The function returns -1 when all command-line arguments
	41	* are parsed. In this case, s->ind is the index of the first non-option
	42	* argument.
	43	*
	44	* @param s status; shall be initialized to KETOPT_INIT on the first call
	45	* @param argc length of argv[]
	46	* @param argv list of command-line arguments; argv[0] is ignored
	47	* @param permute non-zero to move options ahead of non-option arguments
	48	* @param ostr option string
	49	* @param longopts long options
	50	*
	51	* @return ASCII for a short option; ko_longopt_t::val for a long option; -1 if
	52	* argv[] is fully processed; '?' for an unknown option or an ambiguous
	53	* long option; ':' if an option argument is missing
	54	*/
	55	static int ketopt(ketopt_t s, int argc, char argv[], int permute, const char ostr, const ko_longopt_t longopts)
	56	{
	57	int opt = -1, i0, j;
	58	if (permute) {
	59	while (s->i < argc && (argv[s->i][0] != '-' \|\| argv[s->i][1] == '\0'))
	60	++s->i, ++s->n_args;
	61	}
	62	s->arg = 0, s->longidx = -1, i0 = s->i;
	63	if (s->i >= argc \|\| argv[s->i][0] != '-' \|\| argv[s->i][1] == '\0') {
	64	s->ind = s->i - s->n_args;
	65	return -1;
	66	}
	67	if (argv[s->i][0] == '-' && argv[s->i][1] == '-') { /* "--" or a long option */
	68	if (argv[s->i][2] == '\0') { /* a bare "--" */
	69	ketopt_permute(argv, s->i, s->n_args);
	70	++s->i, s->ind = s->i - s->n_args;
	71	return -1;
	72	}
	73	s->opt = 0, opt = '?', s->pos = -1;
	74	if (longopts) { /* parse long options */
	75	int k, n_exact = 0, n_partial = 0;
	76	const ko_longopt_t o = 0, o_exact = 0, *o_partial = 0;
	77	for (j = 2; argv[s->i][j] != '\0' && argv[s->i][j] != '='; ++j) {} /* find the end of the option name */
	78	for (k = 0; longopts[k].name != 0; ++k)
	79	if (strncmp(&argv[s->i][2], longopts[k].name, j - 2) == 0) {
	80	if (longopts[k].name[j - 2] == 0) ++n_exact, o_exact = &longopts[k];
	81	else ++n_partial, o_partial = &longopts[k];
	82	}
	83	if (n_exact > 1 \|\| (n_exact == 0 && n_partial > 1)) return '?';
	84	o = n_exact == 1? o_exact : n_partial == 1? o_partial : 0;
	85	if (o) {
	86	s->opt = opt = o->val, s->longidx = o - longopts;
	87	if (argv[s->i][j] == '=') s->arg = &argv[s->i][j + 1];
	88	if (o->has_arg == 1 && argv[s->i][j] == '\0') {
	89	if (s->i < argc - 1) s->arg = argv[++s->i];
	90	else opt = ':'; /* missing option argument */
	91	}
	92	}
	93	}
	94	} else { /* a short option */
	95	const char *p;
	96	if (s->pos == 0) s->pos = 1;
	97	opt = s->opt = argv[s->i][s->pos++];
	98	p = strchr((char*)ostr, opt);
	99	if (p == 0) {
	100	opt = '?'; /* unknown option */
	101	} else if (p[1] == ':') {
	102	if (argv[s->i][s->pos] == 0) {
	103	if (s->i < argc - 1) s->arg = argv[++s->i];
	104	else opt = ':'; /* missing option argument */
	105	} else s->arg = &argv[s->i][s->pos];
	106	s->pos = -1;
	107	}
	108	}
	109	if (s->pos < 0 \|\| argv[s->i][s->pos] == 0) {
	110	++s->i, s->pos = 0;
	111	if (s->n_args > 0) /* permute */
	112	for (j = i0; j < s->i; ++j)
	113	ketopt_permute(argv, j, s->n_args);
	114	}
	115	s->ind = s->i - s->n_args;
	116	return opt;
	117	}
	118
	119	#endif

+375

-0

backend/lib/mpack-config.h less more

	0	/**
	1	* @defgroup config Configuration Options
	2	*
	3	* Defines the MPack configuration options. You can configure MPack by
	4	* pre-defining any of the below options in your build system or project
	5	* settings.
	6	*
	7	* Custom configuration of MPack is not usually necessary. In almost all
	8	* cases you can ignore this and use the defaults. If you are using the
	9	* amalgamation package, you do not need to add @c mpack-defaults.h to your
	10	* project.
	11	*
	12	* If you do want to configure MPack, the easiest way is to pre-define some of
	13	* the below options as part of your build system or project settings. This
	14	* will override the below defaults.
	15	*
	16	* If you'd like to use a file for configuration instead, define
	17	* @ref MPACK_HAS_CONFIG to 1 in your build system or project settings.
	18	* This will cause MPack to include a file you create called @c mpack-config.h.
	19	* You can copy @c mpack-defaults.h to @c mpack-config.h and make your
	20	* changes, or create a blank @c mpack-config.h and set only the options you
	21	* want. The below settings are the defaults if they are not set by your
	22	* configuration file.
	23	*
	24	* @warning The value of all configuration options must be the same in
	25	* all translation units of your project, as well as in @c mpack.c itself.
	26	* These configuration options affect the layout of structs, among other
	27	* things, which cannot be different in source files that are linked
	28	* together.
	29	*
	30	* @{
	31	*/
	32
	33
	34	/**
	35	* @name Features
	36	* @{
	37	*/
	38
	39	#include <engine.h>
	40
	41	#define MPACK_MALLOC blb_malloc
	42	#define MPACK_REALLOC blb_realloc
	43	#define MPACK_FREE blb_free
	44
	45	/**
	46	* @def MPACK_READER
	47	*
	48	* Enables compilation of the base Tag Reader.
	49	*/
	50	#ifndef MPACK_READER
	51	#define MPACK_READER 1
	52	#endif
	53
	54	/**
	55	* @def MPACK_EXPECT
	56	*
	57	* Enables compilation of the static Expect API.
	58	*/
	59	#ifndef MPACK_EXPECT
	60	#define MPACK_EXPECT 1
	61	#endif
	62
	63	/**
	64	* @def MPACK_NODE
	65	*
	66	* Enables compilation of the dynamic Node API.
	67	*/
	68	#ifndef MPACK_NODE
	69	#define MPACK_NODE 1
	70	#endif
	71
	72	/**
	73	* @def MPACK_WRITER
	74	*
	75	* Enables compilation of the Writer.
	76	*/
	77	#ifndef MPACK_WRITER
	78	#define MPACK_WRITER 1
	79	#endif
	80
	81	/**
	82	* @def MPACK_COMPATIBILITY
	83	*
	84	* Enables compatibility features for reading and writing older
	85	* versions of MessagePack.
	86	*
	87	* This is disabled by default. When disabled, the behaviour is equivalent to
	88	* using the default version, @ref mpack_version_current.
	89	*
	90	* Enable this if you need to interoperate with applications or data that do
	91	* not support the new (v5) MessagePack spec. See the section on v4
	92	* compatibility in @ref docs/protocol.md for more information.
	93	*/
	94	#ifndef MPACK_COMPATIBILITY
	95	#define MPACK_COMPATIBILITY 0
	96	#endif
	97
	98	/**
	99	* @def MPACK_EXTENSIONS
	100	*
	101	* Enables the use of extension types.
	102	*
	103	* This is disabled by default. Define it to 1 to enable it. If disabled,
	104	* functions to read and write extensions will not exist, and any occurrence of
	105	* extension types in parsed messages will flag @ref mpack_error_invalid.
	106	*
	107	* MPack discourages the use of extension types. See the section on extension
	108	* types in @ref docs/protocol.md for more information.
	109	*/
	110	#ifndef MPACK_EXTENSIONS
	111	#define MPACK_EXTENSIONS 1
	112	#endif
	113
	114
	115	/**
	116	* @}
	117	*/
	118
	119
	120	/**
	121	* @name Dependencies
	122	* @{
	123	*/
	124
	125	/**
	126	* @def MPACK_HAS_CONFIG
	127	*
	128	* Enables the use of an @c mpack-config.h configuration file for MPack.
	129	* This file must be in the same folder as @c mpack.h, or it must be
	130	* available from your project's include paths.
	131	*/
	132	// This goes in your project settings.
	133
	134	/**
	135	* @def MPACK_STDLIB
	136	*
	137	* Enables the use of C stdlib. This allows the library to use malloc
	138	* for debugging and in allocation helpers.
	139	*/
	140	#ifndef MPACK_STDLIB
	141	#define MPACK_STDLIB 1
	142	#endif
	143
	144	/**
	145	* @def MPACK_STDIO
	146	*
	147	* Enables the use of C stdio. This adds helpers for easily
	148	* reading/writing C files and makes debugging easier.
	149	*/
	150	#ifndef MPACK_STDIO
	151	#define MPACK_STDIO 1
	152	#endif
	153
	154	/**
	155	* @}
	156	*/
	157
	158
	159	/**
	160	* @name System Functions
	161	* @{
	162	*/
	163
	164	/**
	165	* @def MPACK_MALLOC
	166	*
	167	* Defines the memory allocation function used by MPack. This is used by
	168	* helpers for automatically allocating data the correct size, and for
	169	* debugging functions. If this macro is undefined, the allocation helpers
	170	* will not be compiled.
	171	*
	172	* The default is @c malloc() if @ref MPACK_STDLIB is enabled.
	173	*/
	174	/**
	175	* @def MPACK_FREE
	176	*
	177	* Defines the memory free function used by MPack. This is used by helpers
	178	* for automatically allocating data the correct size. If this macro is
	179	* undefined, the allocation helpers will not be compiled.
	180	*
	181	* The default is @c free() if @ref MPACK_MALLOC has not been customized and
	182	* @ref MPACK_STDLIB is enabled.
	183	*/
	184	/**
	185	* @def MPACK_REALLOC
	186	*
	187	* Defines the realloc function used by MPack. It is used by growable
	188	* buffers to resize more efficiently.
	189	*
	190	* The default is @c realloc() if @ref MPACK_MALLOC has not been customized and
	191	* @ref MPACK_STDLIB is enabled.
	192	*
	193	* This is optional, even when @ref MPACK_MALLOC is used. If @ref MPACK_MALLOC is
	194	* set and @ref MPACK_REALLOC is not, @ref MPACK_MALLOC is used with a simple copy
	195	* to grow buffers.
	196	*/
	197	#if defined(MPACK_STDLIB) && MPACK_STDLIB && !defined(MPACK_MALLOC)
	198	#define MPACK_MALLOC malloc
	199	#define MPACK_REALLOC realloc
	200	#define MPACK_FREE free
	201	#endif
	202
	203	/**
	204	* @}
	205	*/
	206
	207
	208	/**
	209	* @name Debugging Options
	210	*/
	211
	212	/**
	213	* @def MPACK_DEBUG
	214	*
	215	* Enables debug features. You may want to wrap this around your
	216	* own debug preprocs. By default, this is enabled if @c DEBUG or @c _DEBUG
	217	* are defined. (@c NDEBUG is not used since it is allowed to have
	218	* different values in different translation units.)
	219	*/
	220	#if !defined(MPACK_DEBUG) && (defined(DEBUG) \|\| defined(_DEBUG))
	221	#define MPACK_DEBUG 1
	222	#endif
	223
	224	/**
	225	* @def MPACK_STRINGS
	226	*
	227	* Enables descriptive error and type strings.
	228	*
	229	* This can be turned off (by defining it to 0) to maximize space savings
	230	* on embedded devices. If this is disabled, string functions such as
	231	* mpack_error_to_string() and mpack_type_to_string() return an empty string.
	232	*/
	233	#ifndef MPACK_STRINGS
	234	#define MPACK_STRINGS 1
	235	#endif
	236
	237	/**
	238	* Set this to 1 to implement a custom @ref mpack_assert_fail() function.
	239	* See the documentation on @ref mpack_assert_fail() for details.
	240	*
	241	* Asserts are only used when @ref MPACK_DEBUG is enabled, and can be
	242	* triggered by bugs in MPack or bugs due to incorrect usage of MPack.
	243	*/
	244	#ifndef MPACK_CUSTOM_ASSERT
	245	#define MPACK_CUSTOM_ASSERT 0
	246	#endif
	247
	248	/**
	249	* @def MPACK_READ_TRACKING
	250	*
	251	* Enables compound type size tracking for readers. This ensures that the
	252	* correct number of elements or bytes are read from a compound type.
	253	*
	254	* This is enabled by default in debug builds (provided a @c malloc() is
	255	* available.)
	256	*/
	257	#if !defined(MPACK_READ_TRACKING) && \
	258	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	259	defined(MPACK_READER) && MPACK_READER && \
	260	defined(MPACK_MALLOC)
	261	#define MPACK_READ_TRACKING 1
	262	#endif
	263
	264	/**
	265	* @def MPACK_WRITE_TRACKING
	266	*
	267	* Enables compound type size tracking for writers. This ensures that the
	268	* correct number of elements or bytes are written in a compound type.
	269	*
	270	* Note that without write tracking enabled, it is possible for buggy code
	271	* to emit invalid MessagePack without flagging an error by writing the wrong
	272	* number of elements or bytes in a compound type. With tracking enabled,
	273	* MPack will catch such errors and break on the offending line of code.
	274	*
	275	* This is enabled by default in debug builds (provided a @c malloc() is
	276	* available.)
	277	*/
	278	#if !defined(MPACK_WRITE_TRACKING) && \
	279	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	280	defined(MPACK_WRITER) && MPACK_WRITER && \
	281	defined(MPACK_MALLOC)
	282	#define MPACK_WRITE_TRACKING 1
	283	#endif
	284
	285	/**
	286	* @}
	287	*/
	288
	289
	290	/**
	291	* @name Miscellaneous Options
	292	* @{
	293	*/
	294
	295	/**
	296	* Whether to optimize for size or speed.
	297	*
	298	* Optimizing for size simplifies some parsing and encoding algorithms
	299	* at the expense of speed, and saves a few kilobytes of space in the
	300	* resulting executable.
	301	*
	302	* This automatically detects -Os with GCC/Clang. Unfortunately there
	303	* doesn't seem to be a macro defined for /Os under MSVC.
	304	*/
	305	#ifndef MPACK_OPTIMIZE_FOR_SIZE
	306	#ifdef __OPTIMIZE_SIZE__
	307	#define MPACK_OPTIMIZE_FOR_SIZE 1
	308	#else
	309	#define MPACK_OPTIMIZE_FOR_SIZE 0
	310	#endif
	311	#endif
	312
	313	/**
	314	* Stack space in bytes to use when initializing a reader or writer
	315	* with a stack-allocated buffer.
	316	*/
	317	#ifndef MPACK_STACK_SIZE
	318	#define MPACK_STACK_SIZE 4096
	319	#endif
	320
	321	/**
	322	* Buffer size to use for allocated buffers (such as for a file writer.)
	323	*
	324	* Starting with a single page and growing as needed seems to
	325	* provide the best performance with minimal memory waste.
	326	* Increasing this does not improve performance even when writing
	327	* huge messages.
	328	*/
	329	#ifndef MPACK_BUFFER_SIZE
	330	#define MPACK_BUFFER_SIZE 4096
	331	#endif
	332
	333	/**
	334	* Minimum size of an allocated node page in bytes.
	335	*
	336	* The children for a given compound element must be contiguous, so
	337	* larger pages than this may be allocated as needed. (Safety checks
	338	* exist to prevent malicious data from causing too large allocations.)
	339	*
	340	* See @ref mpack_node_data_t for the size of nodes.
	341	*
	342	* Using as many nodes fit in one memory page seems to provide the
	343	* best performance, and has very little waste when parsing small
	344	* messages.
	345	*/
	346	#ifndef MPACK_NODE_PAGE_SIZE
	347	#define MPACK_NODE_PAGE_SIZE 4096
	348	#endif
	349
	350	/**
	351	* The initial depth for the node parser. When MPACK_MALLOC is available,
	352	* the node parser has no practical depth limit, and it is not recursive
	353	* so there is no risk of overflowing the call stack.
	354	*/
	355	#ifndef MPACK_NODE_INITIAL_DEPTH
	356	#define MPACK_NODE_INITIAL_DEPTH 8
	357	#endif
	358
	359	/**
	360	* The maximum depth for the node parser if @ref MPACK_MALLOC is not available.
	361	*/
	362	#ifndef MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC
	363	#define MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC 32
	364	#endif
	365
	366	/**
	367	* @}
	368	*/
	369
	370
	371	/**
	372	* @}
	373	*/
	374

+6439

-0

backend/lib/mpack.c less more

	0	/**
	1	* The MIT License (MIT)
	2	*
	3	* Copyright (c) 2015-2018 Nicholas Fraser
	4	*
	5	* Permission is hereby granted, free of charge, to any person obtaining a copy
	6	* of this software and associated documentation files (the "Software"), to deal
	7	* in the Software without restriction, including without limitation the rights
	8	* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	9	* copies of the Software, and to permit persons to whom the Software is
	10	* furnished to do so, subject to the following conditions:
	11	*
	12	* The above copyright notice and this permission notice shall be included in all
	13	* copies or substantial portions of the Software.
	14	*
	15	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	16	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	17	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	18	* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	19	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	20	* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
	21	* SOFTWARE.
	22	*
	23	*/
	24
	25	/*
	26	* This is the MPack 1.0 amalgamation package.
	27	*
	28	* http://github.com/ludocode/mpack
	29	*/
	30
	31	#define MPACK_INTERNAL 1
	32	#define MPACK_EMIT_INLINE_DEFS 1
	33
	34	#include "mpack.h"
	35
	36
	37	/* mpack/mpack-platform.c.c */
	38
	39
	40	// We define MPACK_EMIT_INLINE_DEFS and include mpack.h to emit
	41	// standalone definitions of all (non-static) inline functions in MPack.
	42
	43	#define MPACK_INTERNAL 1
	44	#define MPACK_EMIT_INLINE_DEFS 1
	45
	46	/* #include "mpack-platform.h" */
	47	/* #include "mpack.h" */
	48
	49
	50	#if MPACK_DEBUG && MPACK_STDIO
	51	#include <stdarg.h>
	52	#endif
	53
	54
	55
	56	#if MPACK_DEBUG
	57
	58	#if MPACK_STDIO
	59	void mpack_assert_fail_format(const char* format, ...) {
	60	char buffer[512];
	61	va_list args;
	62	va_start(args, format);
	63	vsnprintf(buffer, sizeof(buffer), format, args);
	64	va_end(args);
	65	buffer[sizeof(buffer) - 1] = 0;
	66	mpack_assert_fail_wrapper(buffer);
	67	}
	68
	69	void mpack_break_hit_format(const char* format, ...) {
	70	char buffer[512];
	71	va_list args;
	72	va_start(args, format);
	73	vsnprintf(buffer, sizeof(buffer), format, args);
	74	va_end(args);
	75	buffer[sizeof(buffer) - 1] = 0;
	76	mpack_break_hit(buffer);
	77	}
	78	#endif
	79
	80	#if !MPACK_CUSTOM_ASSERT
	81	void mpack_assert_fail(const char* message) {
	82	MPACK_UNUSED(message);
	83
	84	#if MPACK_STDIO
	85	fprintf(stderr, "%s\n", message);
	86	#endif
	87	}
	88	#endif
	89
	90	// We split the assert failure from the wrapper so that a
	91	// custom assert function can return.
	92	void mpack_assert_fail_wrapper(const char* message) {
	93
	94	#ifdef MPACK_GCOV
	95	// gcov marks even __builtin_unreachable() as an uncovered line. this
	96	// silences it.
	97	(mpack_assert_fail(message), __builtin_unreachable());
	98
	99	#else
	100	mpack_assert_fail(message);
	101
	102	// mpack_assert_fail() is not supposed to return. in case it does, we
	103	// abort.
	104
	105	#if !MPACK_NO_BUILTINS
	106	#if defined(__GNUC__) \|\| defined(__clang__)
	107	__builtin_trap();
	108	#elif defined(WIN32)
	109	__debugbreak();
	110	#endif
	111	#endif
	112
	113	#if (defined(__GNUC__) \|\| defined(__clang__)) && !MPACK_NO_BUILTINS
	114	__builtin_abort();
	115	#elif MPACK_STDLIB
	116	abort();
	117	#endif
	118
	119	MPACK_UNREACHABLE;
	120	#endif
	121	}
	122
	123	#if !MPACK_CUSTOM_BREAK
	124
	125	// If we have a custom assert handler, break wraps it by default.
	126	// This allows users of MPack to only implement mpack_assert_fail() without
	127	// having to worry about the difference between assert and break.
	128	//
	129	// MPACK_CUSTOM_BREAK is available to define a separate break handler
	130	// (which is needed by the unit test suite), but this is not offered in
	131	// mpack-config.h for simplicity.
	132
	133	#if MPACK_CUSTOM_ASSERT
	134	void mpack_break_hit(const char* message) {
	135	mpack_assert_fail_wrapper(message);
	136	}
	137	#else
	138	void mpack_break_hit(const char* message) {
	139	MPACK_UNUSED(message);
	140
	141	#if MPACK_STDIO
	142	fprintf(stderr, "%s\n", message);
	143	#endif
	144
	145	#if defined(__GNUC__) \|\| defined(__clang__) && !MPACK_NO_BUILTINS
	146	__builtin_trap();
	147	#elif defined(WIN32) && !MPACK_NO_BUILTINS
	148	__debugbreak();
	149	#elif MPACK_STDLIB
	150	abort();
	151	#endif
	152	}
	153	#endif
	154
	155	#endif
	156
	157	#endif
	158
	159
	160
	161	// The below are adapted from the C wikibook:
	162	// https://en.wikibooks.org/wiki/C_Programming/Strings
	163
	164	#ifndef mpack_memcmp
	165	int mpack_memcmp(const void* s1, const void* s2, size_t n) {
	166	const unsigned char us1 = (const unsigned char ) s1;
	167	const unsigned char us2 = (const unsigned char ) s2;
	168	while (n-- != 0) {
	169	if (us1 != us2)
	170	return (us1 < us2) ? -1 : +1;
	171	us1++;
	172	us2++;
	173	}
	174	return 0;
	175	}
	176	#endif
	177
	178	#ifndef mpack_memcpy
	179	void* mpack_memcpy(void* MPACK_RESTRICT s1, const void* MPACK_RESTRICT s2, size_t n) {
	180	char* MPACK_RESTRICT dst = (char *)s1;
	181	const char* MPACK_RESTRICT src = (const char *)s2;
	182	while (n-- != 0)
	183	dst++ = src++;
	184	return s1;
	185	}
	186	#endif
	187
	188	#ifndef mpack_memmove
	189	void* mpack_memmove(void* s1, const void* s2, size_t n) {
	190	char p1 = (char )s1;
	191	const char p2 = (const char )s2;
	192	if (p2 < p1 && p1 < p2 + n) {
	193	p2 += n;
	194	p1 += n;
	195	while (n-- != 0)
	196	--p1 = --p2;
	197	} else
	198	while (n-- != 0)
	199	p1++ = p2++;
	200	return s1;
	201	}
	202	#endif
	203
	204	#ifndef mpack_memset
	205	void* mpack_memset(void* s, int c, size_t n) {
	206	unsigned char us = (unsigned char )s;
	207	unsigned char uc = (unsigned char)c;
	208	while (n-- != 0)
	209	*us++ = uc;
	210	return s;
	211	}
	212	#endif
	213
	214	#ifndef mpack_strlen
	215	size_t mpack_strlen(const char* s) {
	216	const char* p = s;
	217	while (*p != '\0')
	218	p++;
	219	return (size_t)(p - s);
	220	}
	221	#endif
	222
	223
	224
	225	#if defined(MPACK_MALLOC) && !defined(MPACK_REALLOC)
	226	void* mpack_realloc(void* old_ptr, size_t used_size, size_t new_size) {
	227	if (new_size == 0) {
	228	if (old_ptr)
	229	MPACK_FREE(old_ptr);
	230	return NULL;
	231	}
	232
	233	void* new_ptr = MPACK_MALLOC(new_size);
	234	if (new_ptr == NULL)
	235	return NULL;
	236
	237	mpack_memcpy(new_ptr, old_ptr, used_size);
	238	MPACK_FREE(old_ptr);
	239	return new_ptr;
	240	}
	241	#endif
	242
	243	/* mpack/mpack-common.c.c */
	244
	245	#define MPACK_INTERNAL 1
	246
	247	/* #include "mpack-common.h" */
	248
	249	#if MPACK_DEBUG && MPACK_STDIO
	250	#include <stdarg.h>
	251	#endif
	252
	253	const char* mpack_error_to_string(mpack_error_t error) {
	254	#if MPACK_STRINGS
	255	switch (error) {
	256	#define MPACK_ERROR_STRING_CASE(e) case e: return #e
	257	MPACK_ERROR_STRING_CASE(mpack_ok);
	258	MPACK_ERROR_STRING_CASE(mpack_error_io);
	259	MPACK_ERROR_STRING_CASE(mpack_error_invalid);
	260	MPACK_ERROR_STRING_CASE(mpack_error_unsupported);
	261	MPACK_ERROR_STRING_CASE(mpack_error_type);
	262	MPACK_ERROR_STRING_CASE(mpack_error_too_big);
	263	MPACK_ERROR_STRING_CASE(mpack_error_memory);
	264	MPACK_ERROR_STRING_CASE(mpack_error_bug);
	265	MPACK_ERROR_STRING_CASE(mpack_error_data);
	266	MPACK_ERROR_STRING_CASE(mpack_error_eof);
	267	#undef MPACK_ERROR_STRING_CASE
	268	}
	269	mpack_assert(0, "unrecognized error %i", (int)error);
	270	return "(unknown mpack_error_t)";
	271	#else
	272	MPACK_UNUSED(error);
	273	return "";
	274	#endif
	275	}
	276
	277	const char* mpack_type_to_string(mpack_type_t type) {
	278	#if MPACK_STRINGS
	279	switch (type) {
	280	#define MPACK_TYPE_STRING_CASE(e) case e: return #e
	281	MPACK_TYPE_STRING_CASE(mpack_type_missing);
	282	MPACK_TYPE_STRING_CASE(mpack_type_nil);
	283	MPACK_TYPE_STRING_CASE(mpack_type_bool);
	284	MPACK_TYPE_STRING_CASE(mpack_type_float);
	285	MPACK_TYPE_STRING_CASE(mpack_type_double);
	286	MPACK_TYPE_STRING_CASE(mpack_type_int);
	287	MPACK_TYPE_STRING_CASE(mpack_type_uint);
	288	MPACK_TYPE_STRING_CASE(mpack_type_str);
	289	MPACK_TYPE_STRING_CASE(mpack_type_bin);
	290	MPACK_TYPE_STRING_CASE(mpack_type_array);
	291	MPACK_TYPE_STRING_CASE(mpack_type_map);
	292	#if MPACK_EXTENSIONS
	293	MPACK_TYPE_STRING_CASE(mpack_type_ext);
	294	#endif
	295	#undef MPACK_TYPE_STRING_CASE
	296	}
	297	mpack_assert(0, "unrecognized type %i", (int)type);
	298	return "(unknown mpack_type_t)";
	299	#else
	300	MPACK_UNUSED(type);
	301	return "";
	302	#endif
	303	}
	304
	305	int mpack_tag_cmp(mpack_tag_t left, mpack_tag_t right) {
	306
	307	// positive numbers may be stored as int; convert to uint
	308	if (left.type == mpack_type_int && left.v.i >= 0) {
	309	left.type = mpack_type_uint;
	310	left.v.u = (uint64_t)left.v.i;
	311	}
	312	if (right.type == mpack_type_int && right.v.i >= 0) {
	313	right.type = mpack_type_uint;
	314	right.v.u = (uint64_t)right.v.i;
	315	}
	316
	317	if (left.type != right.type)
	318	return ((int)left.type < (int)right.type) ? -1 : 1;
	319
	320	switch (left.type) {
	321	case mpack_type_missing: // fallthrough
	322	case mpack_type_nil:
	323	return 0;
	324
	325	case mpack_type_bool:
	326	return (int)left.v.b - (int)right.v.b;
	327
	328	case mpack_type_int:
	329	if (left.v.i == right.v.i)
	330	return 0;
	331	return (left.v.i < right.v.i) ? -1 : 1;
	332
	333	case mpack_type_uint:
	334	if (left.v.u == right.v.u)
	335	return 0;
	336	return (left.v.u < right.v.u) ? -1 : 1;
	337
	338	case mpack_type_array:
	339	case mpack_type_map:
	340	if (left.v.n == right.v.n)
	341	return 0;
	342	return (left.v.n < right.v.n) ? -1 : 1;
	343
	344	case mpack_type_str:
	345	case mpack_type_bin:
	346	if (left.v.l == right.v.l)
	347	return 0;
	348	return (left.v.l < right.v.l) ? -1 : 1;
	349
	350	#if MPACK_EXTENSIONS
	351	case mpack_type_ext:
	352	if (left.exttype == right.exttype) {
	353	if (left.v.l == right.v.l)
	354	return 0;
	355	return (left.v.l < right.v.l) ? -1 : 1;
	356	}
	357	return (int)left.exttype - (int)right.exttype;
	358	#endif
	359
	360	// floats should not normally be compared for equality. we compare
	361	// with memcmp() to silence compiler warnings, but this will return
	362	// equal if both are NaNs with the same representation (though we may
	363	// want this, for instance if you are for some bizarre reason using
	364	// floats as map keys.) i'm not sure what the right thing to
	365	// do is here. check for NaN first? always return false if the type
	366	// is float? use operator== and pragmas to silence compiler warning?
	367	// please send me your suggestions.
	368	// note also that we don't convert floats to doubles, so when this is
	369	// used for ordering purposes, all floats are ordered before all
	370	// doubles.
	371	case mpack_type_float:
	372	return mpack_memcmp(&left.v.f, &right.v.f, sizeof(left.v.f));
	373	case mpack_type_double:
	374	return mpack_memcmp(&left.v.d, &right.v.d, sizeof(left.v.d));
	375	}
	376
	377	mpack_assert(0, "unrecognized type %i", (int)left.type);
	378	return false;
	379	}
	380
	381	#if MPACK_DEBUG && MPACK_STDIO
	382	static char mpack_hex_char(uint8_t hex_value) {
	383	return (hex_value < 10) ? (char)('0' + hex_value) : (char)('a' + (hex_value - 10));
	384	}
	385
	386	static void mpack_tag_debug_complete_bin_ext(mpack_tag_t tag, size_t string_length, char* buffer, size_t buffer_size,
	387	const char* prefix, size_t prefix_size)
	388	{
	389	// If at any point in this function we run out of space in the buffer, we
	390	// bail out. The outer tag print wrapper will make sure we have a
	391	// null-terminator.
	392
	393	if (string_length == 0 \|\| string_length >= buffer_size)
	394	return;
	395	buffer += string_length;
	396	buffer_size -= string_length;
	397
	398	size_t total = mpack_tag_bytes(&tag);
	399	if (total == 0) {
	400	strncpy(buffer, ">", buffer_size);
	401	return;
	402	}
	403
	404	strncpy(buffer, ": ", buffer_size);
	405	if (buffer_size < 2)
	406	return;
	407	buffer += 2;
	408	buffer_size -= 2;
	409
	410	size_t hex_bytes = 0;
	411	for (size_t i = 0; i < MPACK_PRINT_BYTE_COUNT && i < prefix_size && buffer_size > 2; ++i) {
	412	uint8_t byte = (uint8_t)prefix[i];
	413	buffer[0] = mpack_hex_char((uint8_t)(byte >> 4));
	414	buffer[1] = mpack_hex_char((uint8_t)(byte & 0xfu));
	415	buffer += 2;
	416	buffer_size -= 2;
	417	++hex_bytes;
	418	}
	419
	420	if (buffer_size != 0)
	421	mpack_snprintf(buffer, buffer_size, "%s>", (total > hex_bytes) ? "..." : "");
	422	}
	423
	424	static void mpack_tag_debug_pseudo_json_bin(mpack_tag_t tag, char* buffer, size_t buffer_size,
	425	const char* prefix, size_t prefix_size)
	426	{
	427	mpack_assert(mpack_tag_type(&tag) == mpack_type_bin);
	428	size_t length = (size_t)mpack_snprintf(buffer, buffer_size, "<binary data of length %u", tag.v.l);
	429	mpack_tag_debug_complete_bin_ext(tag, length, buffer, buffer_size, prefix, prefix_size);
	430	}
	431
	432	#if MPACK_EXTENSIONS
	433	static void mpack_tag_debug_pseudo_json_ext(mpack_tag_t tag, char* buffer, size_t buffer_size,
	434	const char* prefix, size_t prefix_size)
	435	{
	436	mpack_assert(mpack_tag_type(&tag) == mpack_type_ext);
	437	size_t length = (size_t)mpack_snprintf(buffer, buffer_size, "<ext data of type %i and length %u",
	438	mpack_tag_ext_exttype(&tag), mpack_tag_ext_length(&tag));
	439	mpack_tag_debug_complete_bin_ext(tag, length, buffer, buffer_size, prefix, prefix_size);
	440	}
	441	#endif
	442
	443	static void mpack_tag_debug_pseudo_json_impl(mpack_tag_t tag, char* buffer, size_t buffer_size,
	444	const char* prefix, size_t prefix_size)
	445	{
	446	switch (tag.type) {
	447	case mpack_type_missing:
	448	mpack_snprintf(buffer, buffer_size, "<missing!>");
	449	return;
	450	case mpack_type_nil:
	451	mpack_snprintf(buffer, buffer_size, "null");
	452	return;
	453	case mpack_type_bool:
	454	mpack_snprintf(buffer, buffer_size, tag.v.b ? "true" : "false");
	455	return;
	456	case mpack_type_int:
	457	mpack_snprintf(buffer, buffer_size, "%" PRIi64, tag.v.i);
	458	return;
	459	case mpack_type_uint:
	460	mpack_snprintf(buffer, buffer_size, "%" PRIu64, tag.v.u);
	461	return;
	462	case mpack_type_float:
	463	mpack_snprintf(buffer, buffer_size, "%f", tag.v.f);
	464	return;
	465	case mpack_type_double:
	466	mpack_snprintf(buffer, buffer_size, "%f", tag.v.d);
	467	return;
	468
	469	case mpack_type_str:
	470	mpack_snprintf(buffer, buffer_size, "<string of %u bytes>", tag.v.l);
	471	return;
	472	case mpack_type_bin:
	473	mpack_tag_debug_pseudo_json_bin(tag, buffer, buffer_size, prefix, prefix_size);
	474	return;
	475	#if MPACK_EXTENSIONS
	476	case mpack_type_ext:
	477	mpack_tag_debug_pseudo_json_ext(tag, buffer, buffer_size, prefix, prefix_size);
	478	return;
	479	#endif
	480
	481	case mpack_type_array:
	482	mpack_snprintf(buffer, buffer_size, "<array of %u elements>", tag.v.n);
	483	return;
	484	case mpack_type_map:
	485	mpack_snprintf(buffer, buffer_size, "<map of %u key-value pairs>", tag.v.n);
	486	return;
	487	}
	488
	489	mpack_snprintf(buffer, buffer_size, "<unknown!>");
	490	}
	491
	492	void mpack_tag_debug_pseudo_json(mpack_tag_t tag, char* buffer, size_t buffer_size,
	493	const char* prefix, size_t prefix_size)
	494	{
	495	mpack_assert(buffer_size > 0, "buffer size cannot be zero!");
	496	buffer[0] = 0;
	497
	498	mpack_tag_debug_pseudo_json_impl(tag, buffer, buffer_size, prefix, prefix_size);
	499
	500	// We always null-terminate the buffer manually just in case the snprintf()
	501	// function doesn't null-terminate when the string doesn't fit.
	502	buffer[buffer_size - 1] = 0;
	503	}
	504
	505	static void mpack_tag_debug_describe_impl(mpack_tag_t tag, char* buffer, size_t buffer_size) {
	506	switch (tag.type) {
	507	case mpack_type_missing:
	508	mpack_snprintf(buffer, buffer_size, "missing");
	509	return;
	510	case mpack_type_nil:
	511	mpack_snprintf(buffer, buffer_size, "nil");
	512	return;
	513	case mpack_type_bool:
	514	mpack_snprintf(buffer, buffer_size, tag.v.b ? "true" : "false");
	515	return;
	516	case mpack_type_int:
	517	mpack_snprintf(buffer, buffer_size, "int %" PRIi64, tag.v.i);
	518	return;
	519	case mpack_type_uint:
	520	mpack_snprintf(buffer, buffer_size, "uint %" PRIu64, tag.v.u);
	521	return;
	522	case mpack_type_float:
	523	mpack_snprintf(buffer, buffer_size, "float %f", tag.v.f);
	524	return;
	525	case mpack_type_double:
	526	mpack_snprintf(buffer, buffer_size, "double %f", tag.v.d);
	527	return;
	528	case mpack_type_str:
	529	mpack_snprintf(buffer, buffer_size, "str of %u bytes", tag.v.l);
	530	return;
	531	case mpack_type_bin:
	532	mpack_snprintf(buffer, buffer_size, "bin of %u bytes", tag.v.l);
	533	return;
	534	#if MPACK_EXTENSIONS
	535	case mpack_type_ext:
	536	mpack_snprintf(buffer, buffer_size, "ext of type %i, %u bytes",
	537	mpack_tag_ext_exttype(&tag), mpack_tag_ext_length(&tag));
	538	return;
	539	#endif
	540	case mpack_type_array:
	541	mpack_snprintf(buffer, buffer_size, "array of %u elements", tag.v.n);
	542	return;
	543	case mpack_type_map:
	544	mpack_snprintf(buffer, buffer_size, "map of %u key-value pairs", tag.v.n);
	545	return;
	546	}
	547
	548	mpack_snprintf(buffer, buffer_size, "unknown!");
	549	}
	550
	551	void mpack_tag_debug_describe(mpack_tag_t tag, char* buffer, size_t buffer_size) {
	552	mpack_assert(buffer_size > 0, "buffer size cannot be zero!");
	553	buffer[0] = 0;
	554
	555	mpack_tag_debug_describe_impl(tag, buffer, buffer_size);
	556
	557	// We always null-terminate the buffer manually just in case the snprintf()
	558	// function doesn't null-terminate when the string doesn't fit.
	559	buffer[buffer_size - 1] = 0;
	560	}
	561	#endif
	562
	563
	564
	565	#if MPACK_READ_TRACKING \|\| MPACK_WRITE_TRACKING
	566
	567	#ifndef MPACK_TRACKING_INITIAL_CAPACITY
	568	// seems like a reasonable number. we grow by doubling, and it only
	569	// needs to be as long as the maximum depth of the message.
	570	#define MPACK_TRACKING_INITIAL_CAPACITY 8
	571	#endif
	572
	573	mpack_error_t mpack_track_init(mpack_track_t* track) {
	574	track->count = 0;
	575	track->capacity = MPACK_TRACKING_INITIAL_CAPACITY;
	576	track->elements = (mpack_track_element_t)MPACK_MALLOC(sizeof(mpack_track_element_t) track->capacity);
	577	if (track->elements == NULL)
	578	return mpack_error_memory;
	579	return mpack_ok;
	580	}
	581
	582	mpack_error_t mpack_track_grow(mpack_track_t* track) {
	583	mpack_assert(track->elements, "null track elements!");
	584	mpack_assert(track->count == track->capacity, "incorrect growing?");
	585
	586	size_t new_capacity = track->capacity * 2;
	587
	588	mpack_track_element_t* new_elements = (mpack_track_element_t*)mpack_realloc(track->elements,
	589	sizeof(mpack_track_element_t) * track->count, sizeof(mpack_track_element_t) * new_capacity);
	590	if (new_elements == NULL)
	591	return mpack_error_memory;
	592
	593	track->elements = new_elements;
	594	track->capacity = new_capacity;
	595	return mpack_ok;
	596	}
	597
	598	mpack_error_t mpack_track_push(mpack_track_t* track, mpack_type_t type, uint64_t count) {
	599	mpack_assert(track->elements, "null track elements!");
	600	mpack_log("track pushing %s count %i\n", mpack_type_to_string(type), (int)count);
	601
	602	// maps have twice the number of elements (key/value pairs)
	603	if (type == mpack_type_map)
	604	count *= 2;
	605
	606	// grow if needed
	607	if (track->count == track->capacity) {
	608	mpack_error_t error = mpack_track_grow(track);
	609	if (error != mpack_ok)
	610	return error;
	611	}
	612
	613	// insert new track
	614	track->elements[track->count].type = type;
	615	track->elements[track->count].left = count;
	616	++track->count;
	617	return mpack_ok;
	618	}
	619
	620	mpack_error_t mpack_track_pop(mpack_track_t* track, mpack_type_t type) {
	621	mpack_assert(track->elements, "null track elements!");
	622	mpack_log("track popping %s\n", mpack_type_to_string(type));
	623
	624	if (track->count == 0) {
	625	mpack_break("attempting to close a %s but nothing was opened!", mpack_type_to_string(type));
	626	return mpack_error_bug;
	627	}
	628
	629	mpack_track_element_t* element = &track->elements[track->count - 1];
	630
	631	if (element->type != type) {
	632	mpack_break("attempting to close a %s but the open element is a %s!",
	633	mpack_type_to_string(type), mpack_type_to_string(element->type));
	634	return mpack_error_bug;
	635	}
	636
	637	if (element->left != 0) {
	638	mpack_break("attempting to close a %s but there are %" PRIu64 " %s left",
	639	mpack_type_to_string(type), element->left,
	640	(type == mpack_type_map \|\| type == mpack_type_array) ? "elements" : "bytes");
	641	return mpack_error_bug;
	642	}
	643
	644	--track->count;
	645	return mpack_ok;
	646	}
	647
	648	mpack_error_t mpack_track_peek_element(mpack_track_t* track, bool read) {
	649	MPACK_UNUSED(read);
	650	mpack_assert(track->elements, "null track elements!");
	651
	652	// if there are no open elements, that's fine, we can read/write elements at will
	653	if (track->count == 0)
	654	return mpack_ok;
	655
	656	mpack_track_element_t* element = &track->elements[track->count - 1];
	657
	658	if (element->type != mpack_type_map && element->type != mpack_type_array) {
	659	mpack_break("elements cannot be %s within an %s", read ? "read" : "written",
	660	mpack_type_to_string(element->type));
	661	return mpack_error_bug;
	662	}
	663
	664	if (element->left == 0) {
	665	mpack_break("too many elements %s for %s", read ? "read" : "written",
	666	mpack_type_to_string(element->type));
	667	return mpack_error_bug;
	668	}
	669
	670	return mpack_ok;
	671	}
	672
	673	mpack_error_t mpack_track_element(mpack_track_t* track, bool read) {
	674	mpack_error_t error = mpack_track_peek_element(track, read);
	675	if (track->count > 0 && error == mpack_ok)
	676	--track->elements[track->count - 1].left;
	677	return error;
	678	}
	679
	680	mpack_error_t mpack_track_bytes(mpack_track_t* track, bool read, uint64_t count) {
	681	MPACK_UNUSED(read);
	682	mpack_assert(track->elements, "null track elements!");
	683
	684	if (track->count == 0) {
	685	mpack_break("bytes cannot be %s with no open bin, str or ext", read ? "read" : "written");
	686	return mpack_error_bug;
	687	}
	688
	689	mpack_track_element_t* element = &track->elements[track->count - 1];
	690
	691	if (element->type == mpack_type_map \|\| element->type == mpack_type_array) {
	692	mpack_break("bytes cannot be %s within an %s", read ? "read" : "written",
	693	mpack_type_to_string(element->type));
	694	return mpack_error_bug;
	695	}
	696
	697	if (element->left < count) {
	698	mpack_break("too many bytes %s for %s", read ? "read" : "written",
	699	mpack_type_to_string(element->type));
	700	return mpack_error_bug;
	701	}
	702
	703	element->left -= count;
	704	return mpack_ok;
	705	}
	706
	707	mpack_error_t mpack_track_str_bytes_all(mpack_track_t* track, bool read, uint64_t count) {
	708	mpack_error_t error = mpack_track_bytes(track, read, count);
	709	if (error != mpack_ok)
	710	return error;
	711
	712	mpack_track_element_t* element = &track->elements[track->count - 1];
	713
	714	if (element->type != mpack_type_str) {
	715	mpack_break("the open type must be a string, not a %s", mpack_type_to_string(element->type));
	716	return mpack_error_bug;
	717	}
	718
	719	if (element->left != 0) {
	720	mpack_break("not all bytes were read; the wrong byte count was requested for a string read.");
	721	return mpack_error_bug;
	722	}
	723
	724	return mpack_ok;
	725	}
	726
	727	mpack_error_t mpack_track_check_empty(mpack_track_t* track) {
	728	if (track->count != 0) {
	729	mpack_break("unclosed %s", mpack_type_to_string(track->elements[0].type));
	730	return mpack_error_bug;
	731	}
	732	return mpack_ok;
	733	}
	734
	735	mpack_error_t mpack_track_destroy(mpack_track_t* track, bool cancel) {
	736	mpack_error_t error = cancel ? mpack_ok : mpack_track_check_empty(track);
	737	if (track->elements) {
	738	MPACK_FREE(track->elements);
	739	track->elements = NULL;
	740	}
	741	return error;
	742	}
	743	#endif
	744
	745
	746
	747	static bool mpack_utf8_check_impl(const uint8_t* str, size_t count, bool allow_null) {
	748	while (count > 0) {
	749	uint8_t lead = str[0];
	750
	751	// NUL
	752	if (!allow_null && lead == '\0') // we don't allow NUL bytes in MPack C-strings
	753	return false;
	754
	755	// ASCII
	756	if (lead <= 0x7F) {
	757	++str;
	758	--count;
	759
	760	// 2-byte sequence
	761	} else if ((lead & 0xE0) == 0xC0) {
	762	if (count < 2) // truncated sequence
	763	return false;
	764
	765	uint8_t cont = str[1];
	766	if ((cont & 0xC0) != 0x80) // not a continuation byte
	767	return false;
	768
	769	str += 2;
	770	count -= 2;
	771
	772	uint32_t z = ((uint32_t)(lead & ~0xE0) << 6) \|
	773	(uint32_t)(cont & ~0xC0);
	774
	775	if (z < 0x80) // overlong sequence
	776	return false;
	777
	778	// 3-byte sequence
	779	} else if ((lead & 0xF0) == 0xE0) {
	780	if (count < 3) // truncated sequence
	781	return false;
	782
	783	uint8_t cont1 = str[1];
	784	if ((cont1 & 0xC0) != 0x80) // not a continuation byte
	785	return false;
	786	uint8_t cont2 = str[2];
	787	if ((cont2 & 0xC0) != 0x80) // not a continuation byte
	788	return false;
	789
	790	str += 3;
	791	count -= 3;
	792
	793	uint32_t z = ((uint32_t)(lead & ~0xF0) << 12) \|
	794	((uint32_t)(cont1 & ~0xC0) << 6) \|
	795	(uint32_t)(cont2 & ~0xC0);
	796
	797	if (z < 0x800) // overlong sequence
	798	return false;
	799	if (z >= 0xD800 && z <= 0xDFFF) // surrogate
	800	return false;
	801
	802	// 4-byte sequence
	803	} else if ((lead & 0xF8) == 0xF0) {
	804	if (count < 4) // truncated sequence
	805	return false;
	806
	807	uint8_t cont1 = str[1];
	808	if ((cont1 & 0xC0) != 0x80) // not a continuation byte
	809	return false;
	810	uint8_t cont2 = str[2];
	811	if ((cont2 & 0xC0) != 0x80) // not a continuation byte
	812	return false;
	813	uint8_t cont3 = str[3];
	814	if ((cont3 & 0xC0) != 0x80) // not a continuation byte
	815	return false;
	816
	817	str += 4;
	818	count -= 4;
	819
	820	uint32_t z = ((uint32_t)(lead & ~0xF8) << 18) \|
	821	((uint32_t)(cont1 & ~0xC0) << 12) \|
	822	((uint32_t)(cont2 & ~0xC0) << 6) \|
	823	(uint32_t)(cont3 & ~0xC0);
	824
	825	if (z < 0x10000) // overlong sequence
	826	return false;
	827	if (z > 0x10FFFF) // codepoint limit
	828	return false;
	829
	830	} else {
	831	return false; // continuation byte without a lead, or lead for a 5-byte sequence or longer
	832	}
	833	}
	834	return true;
	835	}
	836
	837	bool mpack_utf8_check(const char* str, size_t bytes) {
	838	return mpack_utf8_check_impl((const uint8_t*)str, bytes, true);
	839	}
	840
	841	bool mpack_utf8_check_no_null(const char* str, size_t bytes) {
	842	return mpack_utf8_check_impl((const uint8_t*)str, bytes, false);
	843	}
	844
	845	bool mpack_str_check_no_null(const char* str, size_t bytes) {
	846	for (size_t i = 0; i < bytes; ++i)
	847	if (str[i] == '\0')
	848	return false;
	849	return true;
	850	}
	851
	852	#if MPACK_DEBUG && MPACK_STDIO
	853	void mpack_print_append(mpack_print_t* print, const char* data, size_t count) {
	854
	855	// copy whatever fits into the buffer
	856	size_t copy = print->size - print->count;
	857	if (copy > count)
	858	copy = count;
	859	mpack_memcpy(print->buffer + print->count, data, copy);
	860	print->count += copy;
	861	data += copy;
	862	count -= copy;
	863
	864	// if we don't need to flush or can't flush there's nothing else to do
	865	if (count == 0 \|\| print->callback == NULL)
	866	return;
	867
	868	// flush the buffer
	869	print->callback(print->context, print->buffer, print->count);
	870
	871	if (count > print->size / 2) {
	872	// flush the rest of the data
	873	print->count = 0;
	874	print->callback(print->context, data, count);
	875	} else {
	876	// copy the rest of the data into the buffer
	877	mpack_memcpy(print->buffer, data, count);
	878	print->count = count;
	879	}
	880
	881	}
	882
	883	void mpack_print_flush(mpack_print_t* print) {
	884	if (print->count > 0 && print->callback != NULL) {
	885	print->callback(print->context, print->buffer, print->count);
	886	print->count = 0;
	887	}
	888	}
	889
	890	void mpack_print_file_callback(void* context, const char* data, size_t count) {
	891	FILE* file = (FILE*)context;
	892	fwrite(data, 1, count, file);
	893	}
	894	#endif
	895
	896	/* mpack/mpack-writer.c.c */
	897
	898	#define MPACK_INTERNAL 1
	899
	900	/* #include "mpack-writer.h" */
	901
	902	#if MPACK_WRITER
	903
	904	#if MPACK_WRITE_TRACKING
	905	static void mpack_writer_flag_if_error(mpack_writer_t* writer, mpack_error_t error) {
	906	if (error != mpack_ok)
	907	mpack_writer_flag_error(writer, error);
	908	}
	909
	910	void mpack_writer_track_push(mpack_writer_t* writer, mpack_type_t type, uint64_t count) {
	911	if (writer->error == mpack_ok)
	912	mpack_writer_flag_if_error(writer, mpack_track_push(&writer->track, type, count));
	913	}
	914
	915	void mpack_writer_track_pop(mpack_writer_t* writer, mpack_type_t type) {
	916	if (writer->error == mpack_ok)
	917	mpack_writer_flag_if_error(writer, mpack_track_pop(&writer->track, type));
	918	}
	919
	920	void mpack_writer_track_element(mpack_writer_t* writer) {
	921	if (writer->error == mpack_ok)
	922	mpack_writer_flag_if_error(writer, mpack_track_element(&writer->track, false));
	923	}
	924
	925	void mpack_writer_track_bytes(mpack_writer_t* writer, size_t count) {
	926	if (writer->error == mpack_ok)
	927	mpack_writer_flag_if_error(writer, mpack_track_bytes(&writer->track, false, count));
	928	}
	929	#endif
	930
	931	static void mpack_writer_clear(mpack_writer_t* writer) {
	932	#if MPACK_COMPATIBILITY
	933	writer->version = mpack_version_current;
	934	#endif
	935	writer->flush = NULL;
	936	writer->error_fn = NULL;
	937	writer->teardown = NULL;
	938	writer->context = NULL;
	939
	940	writer->buffer = NULL;
	941	writer->current = NULL;
	942	writer->end = NULL;
	943	writer->error = mpack_ok;
	944
	945	#if MPACK_WRITE_TRACKING
	946	mpack_memset(&writer->track, 0, sizeof(writer->track));
	947	#endif
	948	}
	949
	950	void mpack_writer_init(mpack_writer_t* writer, char* buffer, size_t size) {
	951	mpack_assert(buffer != NULL, "cannot initialize writer with empty buffer");
	952	mpack_writer_clear(writer);
	953	writer->buffer = buffer;
	954	writer->current = buffer;
	955	writer->end = writer->buffer + size;
	956
	957	#if MPACK_WRITE_TRACKING
	958	mpack_writer_flag_if_error(writer, mpack_track_init(&writer->track));
	959	#endif
	960
	961	mpack_log("===========================\n");
	962	mpack_log("initializing writer with buffer size %i\n", (int)size);
	963	}
	964
	965	void mpack_writer_init_error(mpack_writer_t* writer, mpack_error_t error) {
	966	mpack_writer_clear(writer);
	967	writer->error = error;
	968
	969	mpack_log("===========================\n");
	970	mpack_log("initializing writer in error state %i\n", (int)error);
	971	}
	972
	973	void mpack_writer_set_flush(mpack_writer_t* writer, mpack_writer_flush_t flush) {
	974	MPACK_STATIC_ASSERT(MPACK_WRITER_MINIMUM_BUFFER_SIZE >= MPACK_MAXIMUM_TAG_SIZE,
	975	"minimum buffer size must fit any tag!");
	976	MPACK_STATIC_ASSERT(31 + MPACK_TAG_SIZE_FIXSTR >= MPACK_WRITER_MINIMUM_BUFFER_SIZE,
	977	"minimum buffer size must fit the largest possible fixstr!");
	978
	979	if (mpack_writer_buffer_size(writer) < MPACK_WRITER_MINIMUM_BUFFER_SIZE) {
	980	mpack_break("buffer size is %i, but minimum buffer size for flush is %i",
	981	(int)mpack_writer_buffer_size(writer), MPACK_WRITER_MINIMUM_BUFFER_SIZE);
	982	mpack_writer_flag_error(writer, mpack_error_bug);
	983	return;
	984	}
	985
	986	writer->flush = flush;
	987	}
	988
	989	#ifdef MPACK_MALLOC
	990	typedef struct mpack_growable_writer_t {
	991	char** target_data;
	992	size_t* target_size;
	993	} mpack_growable_writer_t;
	994
	995	static char* mpack_writer_get_reserved(mpack_writer_t* writer) {
	996	// This is in a separate function in order to avoid false strict aliasing
	997	// warnings. We aren't actually violating strict aliasing (the reserved
	998	// space is only ever dereferenced as an mpack_growable_writer_t.)
	999	return (char*)writer->reserved;
	1000	}
	1001
	1002	static void mpack_growable_writer_flush(mpack_writer_t* writer, const char* data, size_t count) {
	1003
	1004	// This is an intrusive flush function which modifies the writer's buffer
	1005	// in response to a flush instead of emptying it in order to add more
	1006	// capacity for data. This removes the need to copy data from a fixed buffer
	1007	// into a growable one, improving performance.
	1008	//
	1009	// There are three ways flush can be called:
	1010	// - flushing the buffer during writing (used is zero, count is all data, data is buffer)
	1011	// - flushing extra data during writing (used is all flushed data, count is extra data, data is not buffer)
	1012	// - flushing during teardown (used and count are both all flushed data, data is buffer)
	1013	//
	1014	// In the first two cases, we grow the buffer by at least double, enough
	1015	// to ensure that new data will fit. We ignore the teardown flush.
	1016
	1017	if (data == writer->buffer) {
	1018
	1019	// teardown, do nothing
	1020	if (mpack_writer_buffer_used(writer) == count)
	1021	return;
	1022
	1023	// otherwise leave the data in the buffer and just grow
	1024	writer->current = writer->buffer + count;
	1025	count = 0;
	1026	}
	1027
	1028	size_t used = mpack_writer_buffer_used(writer);
	1029	size_t size = mpack_writer_buffer_size(writer);
	1030
	1031	mpack_log("flush size %i used %i data %p buffer %p\n",
	1032	(int)count, (int)used, data, writer->buffer);
	1033
	1034	mpack_assert(data == writer->buffer \|\| used + count > size,
	1035	"extra flush for %i but there is %i space left in the buffer! (%i/%i)",
	1036	(int)count, (int)mpack_writer_buffer_left(writer), (int)used, (int)size);
	1037
	1038	// grow to fit the data
	1039	// TODO: this really needs to correctly test for overflow
	1040	size_t new_size = size * 2;
	1041	while (new_size < used + count)
	1042	new_size *= 2;
	1043
	1044	mpack_log("flush growing buffer size from %i to %i\n", (int)size, (int)new_size);
	1045
	1046	// grow the buffer
	1047	char* new_buffer = (char*)mpack_realloc(writer->buffer, used, new_size);
	1048	if (new_buffer == NULL) {
	1049	mpack_writer_flag_error(writer, mpack_error_memory);
	1050	return;
	1051	}
	1052	writer->current = new_buffer + used;
	1053	writer->buffer = new_buffer;
	1054	writer->end = writer->buffer + new_size;
	1055
	1056	// append the extra data
	1057	if (count > 0) {
	1058	mpack_memcpy(writer->current, data, count);
	1059	writer->current += count;
	1060	}
	1061
	1062	mpack_log("new buffer %p, used %i\n", new_buffer, (int)mpack_writer_buffer_used(writer));
	1063	}
	1064
	1065	static void mpack_growable_writer_teardown(mpack_writer_t* writer) {
	1066	mpack_growable_writer_t* growable_writer = (mpack_growable_writer_t*)mpack_writer_get_reserved(writer);
	1067
	1068	if (mpack_writer_error(writer) == mpack_ok) {
	1069
	1070	// shrink the buffer to an appropriate size if the data is
	1071	// much smaller than the buffer
	1072	if (mpack_writer_buffer_used(writer) < mpack_writer_buffer_size(writer) / 2) {
	1073	size_t used = mpack_writer_buffer_used(writer);
	1074
	1075	// We always return a non-null pointer that must be freed, even if
	1076	// nothing was written. malloc() and realloc() do not necessarily
	1077	// do this so we enforce it ourselves.
	1078	size_t size = (used != 0) ? used : 1;
	1079
	1080	char* buffer = (char*)mpack_realloc(writer->buffer, used, size);
	1081	if (!buffer) {
	1082	MPACK_FREE(writer->buffer);
	1083	mpack_writer_flag_error(writer, mpack_error_memory);
	1084	return;
	1085	}
	1086	writer->buffer = buffer;
	1087	writer->end = (writer->current = writer->buffer + used);
	1088	}
	1089
	1090	*growable_writer->target_data = writer->buffer;
	1091	*growable_writer->target_size = mpack_writer_buffer_used(writer);
	1092	writer->buffer = NULL;
	1093
	1094	} else if (writer->buffer) {
	1095	MPACK_FREE(writer->buffer);
	1096	writer->buffer = NULL;
	1097	}
	1098
	1099	writer->context = NULL;
	1100	}
	1101
	1102	void mpack_writer_init_growable(mpack_writer_t* writer, char** target_data, size_t* target_size) {
	1103	mpack_assert(target_data != NULL, "cannot initialize writer without a destination for the data");
	1104	mpack_assert(target_size != NULL, "cannot initialize writer without a destination for the size");
	1105
	1106	*target_data = NULL;
	1107	*target_size = 0;
	1108
	1109	MPACK_STATIC_ASSERT(sizeof(mpack_growable_writer_t) <= sizeof(writer->reserved),
	1110	"not enough reserved space for growable writer!");
	1111	mpack_growable_writer_t* growable_writer = (mpack_growable_writer_t*)mpack_writer_get_reserved(writer);
	1112
	1113	growable_writer->target_data = target_data;
	1114	growable_writer->target_size = target_size;
	1115
	1116	size_t capacity = MPACK_BUFFER_SIZE;
	1117	char* buffer = (char*)MPACK_MALLOC(capacity);
	1118	if (buffer == NULL) {
	1119	mpack_writer_init_error(writer, mpack_error_memory);
	1120	return;
	1121	}
	1122
	1123	mpack_writer_init(writer, buffer, capacity);
	1124	mpack_writer_set_flush(writer, mpack_growable_writer_flush);
	1125	mpack_writer_set_teardown(writer, mpack_growable_writer_teardown);
	1126	}
	1127	#endif
	1128
	1129	#if MPACK_STDIO
	1130	static void mpack_file_writer_flush(mpack_writer_t* writer, const char* buffer, size_t count) {
	1131	FILE* file = (FILE*)writer->context;
	1132	size_t written = fwrite((const void*)buffer, 1, count, file);
	1133	if (written != count)
	1134	mpack_writer_flag_error(writer, mpack_error_io);
	1135	}
	1136
	1137	static void mpack_file_writer_teardown(mpack_writer_t* writer) {
	1138	MPACK_FREE(writer->buffer);
	1139	writer->buffer = NULL;
	1140	writer->context = NULL;
	1141	}
	1142
	1143	static void mpack_file_writer_teardown_close(mpack_writer_t* writer) {
	1144	FILE* file = (FILE*)writer->context;
	1145
	1146	if (file) {
	1147	int ret = fclose(file);
	1148	if (ret != 0)
	1149	mpack_writer_flag_error(writer, mpack_error_io);
	1150	}
	1151
	1152	mpack_file_writer_teardown(writer);
	1153	}
	1154
	1155	void mpack_writer_init_stdfile(mpack_writer_t* writer, FILE* file, bool close_when_done) {
	1156	mpack_assert(file != NULL, "file is NULL");
	1157
	1158	size_t capacity = MPACK_BUFFER_SIZE;
	1159	char* buffer = (char*)MPACK_MALLOC(capacity);
	1160	if (buffer == NULL) {
	1161	mpack_writer_init_error(writer, mpack_error_memory);
	1162	if (close_when_done) {
	1163	fclose(file);
	1164	}
	1165	return;
	1166	}
	1167
	1168	mpack_writer_init(writer, buffer, capacity);
	1169	mpack_writer_set_context(writer, file);
	1170	mpack_writer_set_flush(writer, mpack_file_writer_flush);
	1171	mpack_writer_set_teardown(writer, close_when_done ?
	1172	mpack_file_writer_teardown_close :
	1173	mpack_file_writer_teardown);
	1174	}
	1175
	1176	void mpack_writer_init_filename(mpack_writer_t* writer, const char* filename) {
	1177	mpack_assert(filename != NULL, "filename is NULL");
	1178
	1179	FILE* file = fopen(filename, "wb");
	1180	if (file == NULL) {
	1181	mpack_writer_init_error(writer, mpack_error_io);
	1182	return;
	1183	}
	1184
	1185	mpack_writer_init_stdfile(writer, file, true);
	1186	}
	1187	#endif
	1188
	1189	void mpack_writer_flag_error(mpack_writer_t* writer, mpack_error_t error) {
	1190	mpack_log("writer %p setting error %i: %s\n", writer, (int)error, mpack_error_to_string(error));
	1191
	1192	if (writer->error == mpack_ok) {
	1193	writer->error = error;
	1194	if (writer->error_fn)
	1195	writer->error_fn(writer, writer->error);
	1196	}
	1197	}
	1198
	1199	MPACK_STATIC_INLINE void mpack_writer_flush_unchecked(mpack_writer_t* writer) {
	1200	// This is a bit ugly; we reset used before calling flush so that
	1201	// a flush function can distinguish between flushing the buffer
	1202	// versus flushing external data. see mpack_growable_writer_flush()
	1203	size_t used = mpack_writer_buffer_used(writer);
	1204	writer->current = writer->buffer;
	1205	writer->flush(writer, writer->buffer, used);
	1206	}
	1207
	1208	void mpack_writer_flush_message(mpack_writer_t* writer) {
	1209	if (writer->error != mpack_ok)
	1210	return;
	1211
	1212	#if MPACK_WRITE_TRACKING
	1213	mpack_writer_flag_if_error(writer, mpack_track_check_empty(&writer->track));
	1214	if (writer->error != mpack_ok)
	1215	return;
	1216	#endif
	1217
	1218	if (writer->flush == NULL) {
	1219	mpack_break("cannot call mpack_writer_flush_message() without a flush function!");
	1220	mpack_writer_flag_error(writer, mpack_error_bug);
	1221	return;
	1222	}
	1223
	1224	if (mpack_writer_buffer_used(writer) > 0)
	1225	mpack_writer_flush_unchecked(writer);
	1226	}
	1227
	1228	// Ensures there are at least count bytes free in the buffer. This
	1229	// will flag an error if the flush function fails to make enough
	1230	// room in the buffer.
	1231	MPACK_NOINLINE static bool mpack_writer_ensure(mpack_writer_t* writer, size_t count) {
	1232	mpack_assert(count != 0, "cannot ensure zero bytes!");
	1233	mpack_assert(count <= MPACK_WRITER_MINIMUM_BUFFER_SIZE,
	1234	"cannot ensure %i bytes, this is more than the minimum buffer size %i!",
	1235	(int)count, (int)MPACK_WRITER_MINIMUM_BUFFER_SIZE);
	1236	mpack_assert(count > mpack_writer_buffer_left(writer),
	1237	"request to ensure %i bytes but there are already %i left in the buffer!",
	1238	(int)count, (int)mpack_writer_buffer_left(writer));
	1239
	1240	mpack_log("ensuring %i bytes, %i left\n", (int)count, (int)mpack_writer_buffer_left(writer));
	1241
	1242	if (mpack_writer_error(writer) != mpack_ok)
	1243	return false;
	1244
	1245	if (writer->flush == NULL) {
	1246	mpack_writer_flag_error(writer, mpack_error_too_big);
	1247	return false;
	1248	}
	1249
	1250	mpack_writer_flush_unchecked(writer);
	1251	if (mpack_writer_error(writer) != mpack_ok)
	1252	return false;
	1253
	1254	if (mpack_writer_buffer_left(writer) >= count)
	1255	return true;
	1256
	1257	mpack_writer_flag_error(writer, mpack_error_io);
	1258	return false;
	1259	}
	1260
	1261	// Writes encoded bytes to the buffer when we already know the data
	1262	// does not fit in the buffer (i.e. it straddles the edge of the
	1263	// buffer.) If there is a flush function, it is guaranteed to be
	1264	// called; otherwise mpack_error_too_big is raised.
	1265	MPACK_NOINLINE static void mpack_write_native_straddle(mpack_writer_t* writer, const char* p, size_t count) {
	1266	mpack_assert(count == 0 \|\| p != NULL, "data pointer for %i bytes is NULL", (int)count);
	1267
	1268	if (mpack_writer_error(writer) != mpack_ok)
	1269	return;
	1270	mpack_log("big write for %i bytes from %p, %i space left in buffer\n",
	1271	(int)count, p, (int)mpack_writer_buffer_left(writer));
	1272	mpack_assert(count > mpack_writer_buffer_left(writer),
	1273	"big write requested for %i bytes, but there is %i available "
	1274	"space in buffer. should have called mpack_write_native() instead",
	1275	(int)count, (int)(mpack_writer_buffer_left(writer)));
	1276
	1277	// we'll need a flush function
	1278	if (!writer->flush) {
	1279	mpack_writer_flag_error(writer, mpack_error_too_big);
	1280	return;
	1281	}
	1282
	1283	// flush the buffer
	1284	mpack_writer_flush_unchecked(writer);
	1285	if (mpack_writer_error(writer) != mpack_ok)
	1286	return;
	1287
	1288	// note that an intrusive flush function (such as mpack_growable_writer_flush())
	1289	// may have changed size and/or reset used to a non-zero value. we treat both as
	1290	// though they may have changed, and there may still be data in the buffer.
	1291
	1292	// flush the extra data directly if it doesn't fit in the buffer
	1293	if (count > mpack_writer_buffer_left(writer)) {
	1294	writer->flush(writer, p, count);
	1295	if (mpack_writer_error(writer) != mpack_ok)
	1296	return;
	1297	} else {
	1298	mpack_memcpy(writer->current, p, count);
	1299	writer->current += count;
	1300	}
	1301	}
	1302
	1303	// Writes encoded bytes to the buffer, flushing if necessary.
	1304	MPACK_STATIC_INLINE void mpack_write_native(mpack_writer_t* writer, const char* p, size_t count) {
	1305	mpack_assert(count == 0 \|\| p != NULL, "data pointer for %i bytes is NULL", (int)count);
	1306
	1307	if (mpack_writer_buffer_left(writer) < count) {
	1308	mpack_write_native_straddle(writer, p, count);
	1309	} else {
	1310	mpack_memcpy(writer->current, p, count);
	1311	writer->current += count;
	1312	}
	1313	}
	1314
	1315	mpack_error_t mpack_writer_destroy(mpack_writer_t* writer) {
	1316
	1317	// clean up tracking, asserting if we're not already in an error state
	1318	#if MPACK_WRITE_TRACKING
	1319	mpack_track_destroy(&writer->track, writer->error != mpack_ok);
	1320	#endif
	1321
	1322	// flush any outstanding data
	1323	if (mpack_writer_error(writer) == mpack_ok && mpack_writer_buffer_used(writer) != 0 && writer->flush != NULL) {
	1324	writer->flush(writer, writer->buffer, mpack_writer_buffer_used(writer));
	1325	writer->flush = NULL;
	1326	}
	1327
	1328	if (writer->teardown) {
	1329	writer->teardown(writer);
	1330	writer->teardown = NULL;
	1331	}
	1332
	1333	return writer->error;
	1334	}
	1335
	1336	void mpack_write_tag(mpack_writer_t* writer, mpack_tag_t value) {
	1337	switch (value.type) {
	1338	case mpack_type_missing:
	1339	mpack_break("cannot write a missing value!");
	1340	mpack_writer_flag_error(writer, mpack_error_bug);
	1341	return;
	1342
	1343	case mpack_type_nil: mpack_write_nil (writer); return;
	1344	case mpack_type_bool: mpack_write_bool (writer, value.v.b); return;
	1345	case mpack_type_float: mpack_write_float (writer, value.v.f); return;
	1346	case mpack_type_double: mpack_write_double(writer, value.v.d); return;
	1347	case mpack_type_int: mpack_write_int (writer, value.v.i); return;
	1348	case mpack_type_uint: mpack_write_uint (writer, value.v.u); return;
	1349
	1350	case mpack_type_str: mpack_start_str(writer, value.v.l); return;
	1351	case mpack_type_bin: mpack_start_bin(writer, value.v.l); return;
	1352
	1353	#if MPACK_EXTENSIONS
	1354	case mpack_type_ext:
	1355	mpack_start_ext(writer, mpack_tag_ext_exttype(&value), mpack_tag_ext_length(&value));
	1356	return;
	1357	#endif
	1358
	1359	case mpack_type_array: mpack_start_array(writer, value.v.n); return;
	1360	case mpack_type_map: mpack_start_map(writer, value.v.n); return;
	1361	}
	1362
	1363	mpack_break("unrecognized type %i", (int)value.type);
	1364	mpack_writer_flag_error(writer, mpack_error_bug);
	1365	}
	1366
	1367	MPACK_STATIC_INLINE void mpack_write_byte_element(mpack_writer_t* writer, char value) {
	1368	mpack_writer_track_element(writer);
	1369	if (MPACK_LIKELY(mpack_writer_buffer_left(writer) >= 1) \|\| mpack_writer_ensure(writer, 1))
	1370	*(writer->current++) = value;
	1371	}
	1372
	1373	void mpack_write_nil(mpack_writer_t* writer) {
	1374	mpack_write_byte_element(writer, (char)0xc0);
	1375	}
	1376
	1377	void mpack_write_bool(mpack_writer_t* writer, bool value) {
	1378	mpack_write_byte_element(writer, (char)(0xc2 \| (value ? 1 : 0)));
	1379	}
	1380
	1381	void mpack_write_true(mpack_writer_t* writer) {
	1382	mpack_write_byte_element(writer, (char)0xc3);
	1383	}
	1384
	1385	void mpack_write_false(mpack_writer_t* writer) {
	1386	mpack_write_byte_element(writer, (char)0xc2);
	1387	}
	1388
	1389	void mpack_write_object_bytes(mpack_writer_t* writer, const char* data, size_t bytes) {
	1390	mpack_writer_track_element(writer);
	1391	mpack_write_native(writer, data, bytes);
	1392	}
	1393
	1394	/*
	1395	* Encode functions
	1396	*/
	1397
	1398	MPACK_STATIC_INLINE void mpack_encode_fixuint(char* p, uint8_t value) {
	1399	mpack_assert(value <= 127);
	1400	mpack_store_u8(p, value);
	1401	}
	1402
	1403	MPACK_STATIC_INLINE void mpack_encode_u8(char* p, uint8_t value) {
	1404	mpack_assert(value > 127);
	1405	mpack_store_u8(p, 0xcc);
	1406	mpack_store_u8(p + 1, value);
	1407	}
	1408
	1409	MPACK_STATIC_INLINE void mpack_encode_u16(char* p, uint16_t value) {
	1410	mpack_assert(value > UINT8_MAX);
	1411	mpack_store_u8(p, 0xcd);
	1412	mpack_store_u16(p + 1, value);
	1413	}
	1414
	1415	MPACK_STATIC_INLINE void mpack_encode_u32(char* p, uint32_t value) {
	1416	mpack_assert(value > UINT16_MAX);
	1417	mpack_store_u8(p, 0xce);
	1418	mpack_store_u32(p + 1, value);
	1419	}
	1420
	1421	MPACK_STATIC_INLINE void mpack_encode_u64(char* p, uint64_t value) {
	1422	mpack_assert(value > UINT32_MAX);
	1423	mpack_store_u8(p, 0xcf);
	1424	mpack_store_u64(p + 1, value);
	1425	}
	1426
	1427	MPACK_STATIC_INLINE void mpack_encode_fixint(char* p, int8_t value) {
	1428	// this can encode positive or negative fixints
	1429	mpack_assert(value >= -32);
	1430	mpack_store_i8(p, value);
	1431	}
	1432
	1433	MPACK_STATIC_INLINE void mpack_encode_i8(char* p, int8_t value) {
	1434	mpack_assert(value < -32);
	1435	mpack_store_u8(p, 0xd0);
	1436	mpack_store_i8(p + 1, value);
	1437	}
	1438
	1439	MPACK_STATIC_INLINE void mpack_encode_i16(char* p, int16_t value) {
	1440	mpack_assert(value < INT8_MIN);
	1441	mpack_store_u8(p, 0xd1);
	1442	mpack_store_i16(p + 1, value);
	1443	}
	1444
	1445	MPACK_STATIC_INLINE void mpack_encode_i32(char* p, int32_t value) {
	1446	mpack_assert(value < INT16_MIN);
	1447	mpack_store_u8(p, 0xd2);
	1448	mpack_store_i32(p + 1, value);
	1449	}
	1450
	1451	MPACK_STATIC_INLINE void mpack_encode_i64(char* p, int64_t value) {
	1452	mpack_assert(value < INT32_MIN);
	1453	mpack_store_u8(p, 0xd3);
	1454	mpack_store_i64(p + 1, value);
	1455	}
	1456
	1457	MPACK_STATIC_INLINE void mpack_encode_float(char* p, float value) {
	1458	mpack_store_u8(p, 0xca);
	1459	mpack_store_float(p + 1, value);
	1460	}
	1461
	1462	MPACK_STATIC_INLINE void mpack_encode_double(char* p, double value) {
	1463	mpack_store_u8(p, 0xcb);
	1464	mpack_store_double(p + 1, value);
	1465	}
	1466
	1467	MPACK_STATIC_INLINE void mpack_encode_fixarray(char* p, uint8_t count) {
	1468	mpack_assert(count <= 15);
	1469	mpack_store_u8(p, (uint8_t)(0x90 \| count));
	1470	}
	1471
	1472	MPACK_STATIC_INLINE void mpack_encode_array16(char* p, uint16_t count) {
	1473	mpack_assert(count > 15);
	1474	mpack_store_u8(p, 0xdc);
	1475	mpack_store_u16(p + 1, count);
	1476	}
	1477
	1478	MPACK_STATIC_INLINE void mpack_encode_array32(char* p, uint32_t count) {
	1479	mpack_assert(count > UINT16_MAX);
	1480	mpack_store_u8(p, 0xdd);
	1481	mpack_store_u32(p + 1, count);
	1482	}
	1483
	1484	MPACK_STATIC_INLINE void mpack_encode_fixmap(char* p, uint8_t count) {
	1485	mpack_assert(count <= 15);
	1486	mpack_store_u8(p, (uint8_t)(0x80 \| count));
	1487	}
	1488
	1489	MPACK_STATIC_INLINE void mpack_encode_map16(char* p, uint16_t count) {
	1490	mpack_assert(count > 15);
	1491	mpack_store_u8(p, 0xde);
	1492	mpack_store_u16(p + 1, count);
	1493	}
	1494
	1495	MPACK_STATIC_INLINE void mpack_encode_map32(char* p, uint32_t count) {
	1496	mpack_assert(count > UINT16_MAX);
	1497	mpack_store_u8(p, 0xdf);
	1498	mpack_store_u32(p + 1, count);
	1499	}
	1500
	1501	MPACK_STATIC_INLINE void mpack_encode_fixstr(char* p, uint8_t count) {
	1502	mpack_assert(count <= 31);
	1503	mpack_store_u8(p, (uint8_t)(0xa0 \| count));
	1504	}
	1505
	1506	MPACK_STATIC_INLINE void mpack_encode_str8(char* p, uint8_t count) {
	1507	mpack_assert(count > 31);
	1508	mpack_store_u8(p, 0xd9);
	1509	mpack_store_u8(p + 1, count);
	1510	}
	1511
	1512	MPACK_STATIC_INLINE void mpack_encode_str16(char* p, uint16_t count) {
	1513	// we might be encoding a raw in compatibility mode, so we
	1514	// allow count to be in the range [32, UINT8_MAX].
	1515	mpack_assert(count > 31);
	1516	mpack_store_u8(p, 0xda);
	1517	mpack_store_u16(p + 1, count);
	1518	}
	1519
	1520	MPACK_STATIC_INLINE void mpack_encode_str32(char* p, uint32_t count) {
	1521	mpack_assert(count > UINT16_MAX);
	1522	mpack_store_u8(p, 0xdb);
	1523	mpack_store_u32(p + 1, count);
	1524	}
	1525
	1526	MPACK_STATIC_INLINE void mpack_encode_bin8(char* p, uint8_t count) {
	1527	mpack_store_u8(p, 0xc4);
	1528	mpack_store_u8(p + 1, count);
	1529	}
	1530
	1531	MPACK_STATIC_INLINE void mpack_encode_bin16(char* p, uint16_t count) {
	1532	mpack_assert(count > UINT8_MAX);
	1533	mpack_store_u8(p, 0xc5);
	1534	mpack_store_u16(p + 1, count);
	1535	}
	1536
	1537	MPACK_STATIC_INLINE void mpack_encode_bin32(char* p, uint32_t count) {
	1538	mpack_assert(count > UINT16_MAX);
	1539	mpack_store_u8(p, 0xc6);
	1540	mpack_store_u32(p + 1, count);
	1541	}
	1542
	1543	#if MPACK_EXTENSIONS
	1544	MPACK_STATIC_INLINE void mpack_encode_fixext1(char* p, int8_t exttype) {
	1545	mpack_store_u8(p, 0xd4);
	1546	mpack_store_i8(p + 1, exttype);
	1547	}
	1548
	1549	MPACK_STATIC_INLINE void mpack_encode_fixext2(char* p, int8_t exttype) {
	1550	mpack_store_u8(p, 0xd5);
	1551	mpack_store_i8(p + 1, exttype);
	1552	}
	1553
	1554	MPACK_STATIC_INLINE void mpack_encode_fixext4(char* p, int8_t exttype) {
	1555	mpack_store_u8(p, 0xd6);
	1556	mpack_store_i8(p + 1, exttype);
	1557	}
	1558
	1559	MPACK_STATIC_INLINE void mpack_encode_fixext8(char* p, int8_t exttype) {
	1560	mpack_store_u8(p, 0xd7);
	1561	mpack_store_i8(p + 1, exttype);
	1562	}
	1563
	1564	MPACK_STATIC_INLINE void mpack_encode_fixext16(char* p, int8_t exttype) {
	1565	mpack_store_u8(p, 0xd8);
	1566	mpack_store_i8(p + 1, exttype);
	1567	}
	1568
	1569	MPACK_STATIC_INLINE void mpack_encode_ext8(char* p, int8_t exttype, uint8_t count) {
	1570	mpack_assert(count != 1 && count != 2 && count != 4 && count != 8 && count != 16);
	1571	mpack_store_u8(p, 0xc7);
	1572	mpack_store_u8(p + 1, count);
	1573	mpack_store_i8(p + 2, exttype);
	1574	}
	1575
	1576	MPACK_STATIC_INLINE void mpack_encode_ext16(char* p, int8_t exttype, uint16_t count) {
	1577	mpack_assert(count > UINT8_MAX);
	1578	mpack_store_u8(p, 0xc8);
	1579	mpack_store_u16(p + 1, count);
	1580	mpack_store_i8(p + 3, exttype);
	1581	}
	1582
	1583	MPACK_STATIC_INLINE void mpack_encode_ext32(char* p, int8_t exttype, uint32_t count) {
	1584	mpack_assert(count > UINT16_MAX);
	1585	mpack_store_u8(p, 0xc9);
	1586	mpack_store_u32(p + 1, count);
	1587	mpack_store_i8(p + 5, exttype);
	1588	}
	1589
	1590	MPACK_STATIC_INLINE void mpack_encode_timestamp_4(char* p, uint32_t seconds) {
	1591	mpack_encode_fixext4(p, MPACK_EXTTYPE_TIMESTAMP);
	1592	mpack_store_u32(p + MPACK_TAG_SIZE_FIXEXT4, seconds);
	1593	}
	1594
	1595	MPACK_STATIC_INLINE void mpack_encode_timestamp_8(char* p, int64_t seconds, uint32_t nanoseconds) {
	1596	mpack_assert(nanoseconds <= MPACK_TIMESTAMP_NANOSECONDS_MAX);
	1597	mpack_encode_fixext8(p, MPACK_EXTTYPE_TIMESTAMP);
	1598	uint64_t encoded = ((uint64_t)nanoseconds << 34) \| (uint64_t)seconds;
	1599	mpack_store_u64(p + MPACK_TAG_SIZE_FIXEXT8, encoded);
	1600	}
	1601
	1602	MPACK_STATIC_INLINE void mpack_encode_timestamp_12(char* p, int64_t seconds, uint32_t nanoseconds) {
	1603	mpack_assert(nanoseconds <= MPACK_TIMESTAMP_NANOSECONDS_MAX);
	1604	mpack_encode_ext8(p, MPACK_EXTTYPE_TIMESTAMP, 12);
	1605	mpack_store_u32(p + MPACK_TAG_SIZE_EXT8, nanoseconds);
	1606	mpack_store_i64(p + MPACK_TAG_SIZE_EXT8 + 4, seconds);
	1607	}
	1608	#endif
	1609
	1610
	1611
	1612	/*
	1613	* Write functions
	1614	*/
	1615
	1616	// This is a macro wrapper to the encode functions to encode
	1617	// directly into the buffer. If mpack_writer_ensure() fails
	1618	// it will flag an error so we don't have to do anything.
	1619	#define MPACK_WRITE_ENCODED(encode_fn, size, ...) do { \
	1620	if (MPACK_LIKELY(mpack_writer_buffer_left(writer) >= size) \|\| mpack_writer_ensure(writer, size)) { \
	1621	MPACK_EXPAND(encode_fn(writer->current, __VA_ARGS__)); \
	1622	writer->current += size; \
	1623	} \
	1624	} while (0)
	1625
	1626	void mpack_write_u8(mpack_writer_t* writer, uint8_t value) {
	1627	#if MPACK_OPTIMIZE_FOR_SIZE
	1628	mpack_write_u64(writer, value);
	1629	#else
	1630	mpack_writer_track_element(writer);
	1631	if (value <= 127) {
	1632	MPACK_WRITE_ENCODED(mpack_encode_fixuint, MPACK_TAG_SIZE_FIXUINT, value);
	1633	} else {
	1634	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, value);
	1635	}
	1636	#endif
	1637	}
	1638
	1639	void mpack_write_u16(mpack_writer_t* writer, uint16_t value) {
	1640	#if MPACK_OPTIMIZE_FOR_SIZE
	1641	mpack_write_u64(writer, value);
	1642	#else
	1643	mpack_writer_track_element(writer);
	1644	if (value <= 127) {
	1645	MPACK_WRITE_ENCODED(mpack_encode_fixuint, MPACK_TAG_SIZE_FIXUINT, (uint8_t)value);
	1646	} else if (value <= UINT8_MAX) {
	1647	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1648	} else {
	1649	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, value);
	1650	}
	1651	#endif
	1652	}
	1653
	1654	void mpack_write_u32(mpack_writer_t* writer, uint32_t value) {
	1655	#if MPACK_OPTIMIZE_FOR_SIZE
	1656	mpack_write_u64(writer, value);
	1657	#else
	1658	mpack_writer_track_element(writer);
	1659	if (value <= 127) {
	1660	MPACK_WRITE_ENCODED(mpack_encode_fixuint, MPACK_TAG_SIZE_FIXUINT, (uint8_t)value);
	1661	} else if (value <= UINT8_MAX) {
	1662	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1663	} else if (value <= UINT16_MAX) {
	1664	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, (uint16_t)value);
	1665	} else {
	1666	MPACK_WRITE_ENCODED(mpack_encode_u32, MPACK_TAG_SIZE_U32, value);
	1667	}
	1668	#endif
	1669	}
	1670
	1671	void mpack_write_u64(mpack_writer_t* writer, uint64_t value) {
	1672	mpack_writer_track_element(writer);
	1673
	1674	if (value <= 127) {
	1675	MPACK_WRITE_ENCODED(mpack_encode_fixuint, MPACK_TAG_SIZE_FIXUINT, (uint8_t)value);
	1676	} else if (value <= UINT8_MAX) {
	1677	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1678	} else if (value <= UINT16_MAX) {
	1679	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, (uint16_t)value);
	1680	} else if (value <= UINT32_MAX) {
	1681	MPACK_WRITE_ENCODED(mpack_encode_u32, MPACK_TAG_SIZE_U32, (uint32_t)value);
	1682	} else {
	1683	MPACK_WRITE_ENCODED(mpack_encode_u64, MPACK_TAG_SIZE_U64, value);
	1684	}
	1685	}
	1686
	1687	void mpack_write_i8(mpack_writer_t* writer, int8_t value) {
	1688	#if MPACK_OPTIMIZE_FOR_SIZE
	1689	mpack_write_i64(writer, value);
	1690	#else
	1691	mpack_writer_track_element(writer);
	1692	if (value >= -32) {
	1693	// we encode positive and negative fixints together
	1694	MPACK_WRITE_ENCODED(mpack_encode_fixint, MPACK_TAG_SIZE_FIXINT, (int8_t)value);
	1695	} else {
	1696	MPACK_WRITE_ENCODED(mpack_encode_i8, MPACK_TAG_SIZE_I8, (int8_t)value);
	1697	}
	1698	#endif
	1699	}
	1700
	1701	void mpack_write_i16(mpack_writer_t* writer, int16_t value) {
	1702	#if MPACK_OPTIMIZE_FOR_SIZE
	1703	mpack_write_i64(writer, value);
	1704	#else
	1705	mpack_writer_track_element(writer);
	1706	if (value >= -32) {
	1707	if (value <= 127) {
	1708	// we encode positive and negative fixints together
	1709	MPACK_WRITE_ENCODED(mpack_encode_fixint, MPACK_TAG_SIZE_FIXINT, (int8_t)value);
	1710	} else if (value <= UINT8_MAX) {
	1711	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1712	} else {
	1713	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, (uint16_t)value);
	1714	}
	1715	} else if (value >= INT8_MIN) {
	1716	MPACK_WRITE_ENCODED(mpack_encode_i8, MPACK_TAG_SIZE_I8, (int8_t)value);
	1717	} else {
	1718	MPACK_WRITE_ENCODED(mpack_encode_i16, MPACK_TAG_SIZE_I16, (int16_t)value);
	1719	}
	1720	#endif
	1721	}
	1722
	1723	void mpack_write_i32(mpack_writer_t* writer, int32_t value) {
	1724	#if MPACK_OPTIMIZE_FOR_SIZE
	1725	mpack_write_i64(writer, value);
	1726	#else
	1727	mpack_writer_track_element(writer);
	1728	if (value >= -32) {
	1729	if (value <= 127) {
	1730	// we encode positive and negative fixints together
	1731	MPACK_WRITE_ENCODED(mpack_encode_fixint, MPACK_TAG_SIZE_FIXINT, (int8_t)value);
	1732	} else if (value <= UINT8_MAX) {
	1733	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1734	} else if (value <= UINT16_MAX) {
	1735	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, (uint16_t)value);
	1736	} else {
	1737	MPACK_WRITE_ENCODED(mpack_encode_u32, MPACK_TAG_SIZE_U32, (uint32_t)value);
	1738	}
	1739	} else if (value >= INT8_MIN) {
	1740	MPACK_WRITE_ENCODED(mpack_encode_i8, MPACK_TAG_SIZE_I8, (int8_t)value);
	1741	} else if (value >= INT16_MIN) {
	1742	MPACK_WRITE_ENCODED(mpack_encode_i16, MPACK_TAG_SIZE_I16, (int16_t)value);
	1743	} else {
	1744	MPACK_WRITE_ENCODED(mpack_encode_i32, MPACK_TAG_SIZE_I32, value);
	1745	}
	1746	#endif
	1747	}
	1748
	1749	void mpack_write_i64(mpack_writer_t* writer, int64_t value) {
	1750	#if MPACK_OPTIMIZE_FOR_SIZE
	1751	if (value > 127) {
	1752	// for non-fix positive ints we call the u64 writer to save space
	1753	mpack_write_u64(writer, (uint64_t)value);
	1754	return;
	1755	}
	1756	#endif
	1757
	1758	mpack_writer_track_element(writer);
	1759	if (value >= -32) {
	1760	#if MPACK_OPTIMIZE_FOR_SIZE
	1761	MPACK_WRITE_ENCODED(mpack_encode_fixint, MPACK_TAG_SIZE_FIXINT, (int8_t)value);
	1762	#else
	1763	if (value <= 127) {
	1764	MPACK_WRITE_ENCODED(mpack_encode_fixint, MPACK_TAG_SIZE_FIXINT, (int8_t)value);
	1765	} else if (value <= UINT8_MAX) {
	1766	MPACK_WRITE_ENCODED(mpack_encode_u8, MPACK_TAG_SIZE_U8, (uint8_t)value);
	1767	} else if (value <= UINT16_MAX) {
	1768	MPACK_WRITE_ENCODED(mpack_encode_u16, MPACK_TAG_SIZE_U16, (uint16_t)value);
	1769	} else if (value <= UINT32_MAX) {
	1770	MPACK_WRITE_ENCODED(mpack_encode_u32, MPACK_TAG_SIZE_U32, (uint32_t)value);
	1771	} else {
	1772	MPACK_WRITE_ENCODED(mpack_encode_u64, MPACK_TAG_SIZE_U64, (uint64_t)value);
	1773	}
	1774	#endif
	1775	} else if (value >= INT8_MIN) {
	1776	MPACK_WRITE_ENCODED(mpack_encode_i8, MPACK_TAG_SIZE_I8, (int8_t)value);
	1777	} else if (value >= INT16_MIN) {
	1778	MPACK_WRITE_ENCODED(mpack_encode_i16, MPACK_TAG_SIZE_I16, (int16_t)value);
	1779	} else if (value >= INT32_MIN) {
	1780	MPACK_WRITE_ENCODED(mpack_encode_i32, MPACK_TAG_SIZE_I32, (int32_t)value);
	1781	} else {
	1782	MPACK_WRITE_ENCODED(mpack_encode_i64, MPACK_TAG_SIZE_I64, value);
	1783	}
	1784	}
	1785
	1786	void mpack_write_float(mpack_writer_t* writer, float value) {
	1787	mpack_writer_track_element(writer);
	1788	MPACK_WRITE_ENCODED(mpack_encode_float, MPACK_TAG_SIZE_FLOAT, value);
	1789	}
	1790
	1791	void mpack_write_double(mpack_writer_t* writer, double value) {
	1792	mpack_writer_track_element(writer);
	1793	MPACK_WRITE_ENCODED(mpack_encode_double, MPACK_TAG_SIZE_DOUBLE, value);
	1794	}
	1795
	1796	#if MPACK_EXTENSIONS
	1797	void mpack_write_timestamp(mpack_writer_t* writer, int64_t seconds, uint32_t nanoseconds) {
	1798	#if MPACK_COMPATIBILITY
	1799	if (writer->version <= mpack_version_v4) {
	1800	mpack_break("Timestamps require spec version v5 or later. This writer is in v%i mode.", (int)writer->version);
	1801	mpack_writer_flag_error(writer, mpack_error_bug);
	1802	return;
	1803	}
	1804	#endif
	1805
	1806	if (nanoseconds > MPACK_TIMESTAMP_NANOSECONDS_MAX) {
	1807	mpack_break("timestamp nanoseconds out of bounds: %u", nanoseconds);
	1808	mpack_writer_flag_error(writer, mpack_error_bug);
	1809	return;
	1810	}
	1811
	1812	mpack_writer_track_element(writer);
	1813
	1814	if (seconds < 0 \|\| seconds >= (INT64_C(1) << 34)) {
	1815	MPACK_WRITE_ENCODED(mpack_encode_timestamp_12, MPACK_EXT_SIZE_TIMESTAMP12, seconds, nanoseconds);
	1816	} else if (seconds > UINT32_MAX \|\| nanoseconds > 0) {
	1817	MPACK_WRITE_ENCODED(mpack_encode_timestamp_8, MPACK_EXT_SIZE_TIMESTAMP8, seconds, nanoseconds);
	1818	} else {
	1819	MPACK_WRITE_ENCODED(mpack_encode_timestamp_4, MPACK_EXT_SIZE_TIMESTAMP4, (uint32_t)seconds);
	1820	}
	1821	}
	1822	#endif
	1823
	1824	void mpack_start_array(mpack_writer_t* writer, uint32_t count) {
	1825	mpack_writer_track_element(writer);
	1826
	1827	if (count <= 15) {
	1828	MPACK_WRITE_ENCODED(mpack_encode_fixarray, MPACK_TAG_SIZE_FIXARRAY, (uint8_t)count);
	1829	} else if (count <= UINT16_MAX) {
	1830	MPACK_WRITE_ENCODED(mpack_encode_array16, MPACK_TAG_SIZE_ARRAY16, (uint16_t)count);
	1831	} else {
	1832	MPACK_WRITE_ENCODED(mpack_encode_array32, MPACK_TAG_SIZE_ARRAY32, (uint32_t)count);
	1833	}
	1834
	1835	mpack_writer_track_push(writer, mpack_type_array, count);
	1836	}
	1837
	1838	void mpack_start_map(mpack_writer_t* writer, uint32_t count) {
	1839	mpack_writer_track_element(writer);
	1840
	1841	if (count <= 15) {
	1842	MPACK_WRITE_ENCODED(mpack_encode_fixmap, MPACK_TAG_SIZE_FIXMAP, (uint8_t)count);
	1843	} else if (count <= UINT16_MAX) {
	1844	MPACK_WRITE_ENCODED(mpack_encode_map16, MPACK_TAG_SIZE_MAP16, (uint16_t)count);
	1845	} else {
	1846	MPACK_WRITE_ENCODED(mpack_encode_map32, MPACK_TAG_SIZE_MAP32, (uint32_t)count);
	1847	}
	1848
	1849	mpack_writer_track_push(writer, mpack_type_map, count);
	1850	}
	1851
	1852	static void mpack_start_str_notrack(mpack_writer_t* writer, uint32_t count) {
	1853	if (count <= 31) {
	1854	MPACK_WRITE_ENCODED(mpack_encode_fixstr, MPACK_TAG_SIZE_FIXSTR, (uint8_t)count);
	1855
	1856	// str8 is only supported in v5 or later.
	1857	} else if (count <= UINT8_MAX
	1858	#if MPACK_COMPATIBILITY
	1859	&& writer->version >= mpack_version_v5
	1860	#endif
	1861	) {
	1862	MPACK_WRITE_ENCODED(mpack_encode_str8, MPACK_TAG_SIZE_STR8, (uint8_t)count);
	1863
	1864	} else if (count <= UINT16_MAX) {
	1865	MPACK_WRITE_ENCODED(mpack_encode_str16, MPACK_TAG_SIZE_STR16, (uint16_t)count);
	1866	} else {
	1867	MPACK_WRITE_ENCODED(mpack_encode_str32, MPACK_TAG_SIZE_STR32, (uint32_t)count);
	1868	}
	1869	}
	1870
	1871	static void mpack_start_bin_notrack(mpack_writer_t* writer, uint32_t count) {
	1872	#if MPACK_COMPATIBILITY
	1873	// In the v4 spec, there was only the raw type for any kind of
	1874	// variable-length data. In v4 mode, we support the bin functions,
	1875	// but we produce an old-style raw.
	1876	if (writer->version <= mpack_version_v4) {
	1877	mpack_start_str_notrack(writer, count);
	1878	return;
	1879	}
	1880	#endif
	1881
	1882	if (count <= UINT8_MAX) {
	1883	MPACK_WRITE_ENCODED(mpack_encode_bin8, MPACK_TAG_SIZE_BIN8, (uint8_t)count);
	1884	} else if (count <= UINT16_MAX) {
	1885	MPACK_WRITE_ENCODED(mpack_encode_bin16, MPACK_TAG_SIZE_BIN16, (uint16_t)count);
	1886	} else {
	1887	MPACK_WRITE_ENCODED(mpack_encode_bin32, MPACK_TAG_SIZE_BIN32, (uint32_t)count);
	1888	}
	1889	}
	1890
	1891	void mpack_start_str(mpack_writer_t* writer, uint32_t count) {
	1892	mpack_writer_track_element(writer);
	1893	mpack_start_str_notrack(writer, count);
	1894	mpack_writer_track_push(writer, mpack_type_str, count);
	1895	}
	1896
	1897	void mpack_start_bin(mpack_writer_t* writer, uint32_t count) {
	1898	mpack_writer_track_element(writer);
	1899	mpack_start_bin_notrack(writer, count);
	1900	mpack_writer_track_push(writer, mpack_type_bin, count);
	1901	}
	1902
	1903	#if MPACK_EXTENSIONS
	1904	void mpack_start_ext(mpack_writer_t* writer, int8_t exttype, uint32_t count) {
	1905	#if MPACK_COMPATIBILITY
	1906	if (writer->version <= mpack_version_v4) {
	1907	mpack_break("Ext types require spec version v5 or later. This writer is in v%i mode.", (int)writer->version);
	1908	mpack_writer_flag_error(writer, mpack_error_bug);
	1909	return;
	1910	}
	1911	#endif
	1912
	1913	mpack_writer_track_element(writer);
	1914
	1915	if (count == 1) {
	1916	MPACK_WRITE_ENCODED(mpack_encode_fixext1, MPACK_TAG_SIZE_FIXEXT1, exttype);
	1917	} else if (count == 2) {
	1918	MPACK_WRITE_ENCODED(mpack_encode_fixext2, MPACK_TAG_SIZE_FIXEXT2, exttype);
	1919	} else if (count == 4) {
	1920	MPACK_WRITE_ENCODED(mpack_encode_fixext4, MPACK_TAG_SIZE_FIXEXT4, exttype);
	1921	} else if (count == 8) {
	1922	MPACK_WRITE_ENCODED(mpack_encode_fixext8, MPACK_TAG_SIZE_FIXEXT8, exttype);
	1923	} else if (count == 16) {
	1924	MPACK_WRITE_ENCODED(mpack_encode_fixext16, MPACK_TAG_SIZE_FIXEXT16, exttype);
	1925	} else if (count <= UINT8_MAX) {
	1926	MPACK_WRITE_ENCODED(mpack_encode_ext8, MPACK_TAG_SIZE_EXT8, exttype, (uint8_t)count);
	1927	} else if (count <= UINT16_MAX) {
	1928	MPACK_WRITE_ENCODED(mpack_encode_ext16, MPACK_TAG_SIZE_EXT16, exttype, (uint16_t)count);
	1929	} else {
	1930	MPACK_WRITE_ENCODED(mpack_encode_ext32, MPACK_TAG_SIZE_EXT32, exttype, (uint32_t)count);
	1931	}
	1932
	1933	mpack_writer_track_push(writer, mpack_type_ext, count);
	1934	}
	1935	#endif
	1936
	1937
	1938
	1939	/*
	1940	* Compound helpers and other functions
	1941	*/
	1942
	1943	void mpack_write_str(mpack_writer_t* writer, const char* data, uint32_t count) {
	1944	mpack_assert(data != NULL, "data for string of length %i is NULL", (int)count);
	1945
	1946	#if MPACK_OPTIMIZE_FOR_SIZE
	1947	mpack_writer_track_element(writer);
	1948	mpack_start_str_notrack(writer, count);
	1949	mpack_write_native(writer, data, count);
	1950	#else
	1951
	1952	mpack_writer_track_element(writer);
	1953
	1954	if (count <= 31) {
	1955	// The minimum buffer size when using a flush function is guaranteed to
	1956	// fit the largest possible fixstr.
	1957	size_t size = count + MPACK_TAG_SIZE_FIXSTR;
	1958	if (MPACK_LIKELY(mpack_writer_buffer_left(writer) >= size) \|\| mpack_writer_ensure(writer, size)) {
	1959	char* MPACK_RESTRICT p = writer->current;
	1960	mpack_encode_fixstr(p, (uint8_t)count);
	1961	mpack_memcpy(p + MPACK_TAG_SIZE_FIXSTR, data, count);
	1962	writer->current += count + MPACK_TAG_SIZE_FIXSTR;
	1963	}
	1964	return;
	1965	}
	1966
	1967	if (count <= UINT8_MAX
	1968	#if MPACK_COMPATIBILITY
	1969	&& writer->version >= mpack_version_v5
	1970	#endif
	1971	) {
	1972	if (count + MPACK_TAG_SIZE_STR8 <= mpack_writer_buffer_left(writer)) {
	1973	char* MPACK_RESTRICT p = writer->current;
	1974	mpack_encode_str8(p, (uint8_t)count);
	1975	mpack_memcpy(p + MPACK_TAG_SIZE_STR8, data, count);
	1976	writer->current += count + MPACK_TAG_SIZE_STR8;
	1977	} else {
	1978	MPACK_WRITE_ENCODED(mpack_encode_str8, MPACK_TAG_SIZE_STR8, (uint8_t)count);
	1979	mpack_write_native(writer, data, count);
	1980	}
	1981	return;
	1982	}
	1983
	1984	// str16 and str32 are likely to be a significant fraction of the buffer
	1985	// size, so we don't bother with a combined space check in order to
	1986	// minimize code size.
	1987	if (count <= UINT16_MAX) {
	1988	MPACK_WRITE_ENCODED(mpack_encode_str16, MPACK_TAG_SIZE_STR16, (uint16_t)count);
	1989	mpack_write_native(writer, data, count);
	1990	} else {
	1991	MPACK_WRITE_ENCODED(mpack_encode_str32, MPACK_TAG_SIZE_STR32, (uint32_t)count);
	1992	mpack_write_native(writer, data, count);
	1993	}
	1994
	1995	#endif
	1996	}
	1997
	1998	void mpack_write_bin(mpack_writer_t* writer, const char* data, uint32_t count) {
	1999	mpack_assert(data != NULL, "data pointer for bin of %i bytes is NULL", (int)count);
	2000	mpack_start_bin(writer, count);
	2001	mpack_write_bytes(writer, data, count);
	2002	mpack_finish_bin(writer);
	2003	}
	2004
	2005	#if MPACK_EXTENSIONS
	2006	void mpack_write_ext(mpack_writer_t* writer, int8_t exttype, const char* data, uint32_t count) {
	2007	mpack_assert(data != NULL, "data pointer for ext of type %i and %i bytes is NULL", exttype, (int)count);
	2008	mpack_start_ext(writer, exttype, count);
	2009	mpack_write_bytes(writer, data, count);
	2010	mpack_finish_ext(writer);
	2011	}
	2012	#endif
	2013
	2014	void mpack_write_bytes(mpack_writer_t* writer, const char* data, size_t count) {
	2015	mpack_assert(data != NULL, "data pointer for %i bytes is NULL", (int)count);
	2016	mpack_writer_track_bytes(writer, count);
	2017	mpack_write_native(writer, data, count);
	2018	}
	2019
	2020	void mpack_write_cstr(mpack_writer_t* writer, const char* cstr) {
	2021	mpack_assert(cstr != NULL, "cstr pointer is NULL");
	2022	size_t length = mpack_strlen(cstr);
	2023	if (length > UINT32_MAX)
	2024	mpack_writer_flag_error(writer, mpack_error_invalid);
	2025	mpack_write_str(writer, cstr, (uint32_t)length);
	2026	}
	2027
	2028	void mpack_write_cstr_or_nil(mpack_writer_t* writer, const char* cstr) {
	2029	if (cstr)
	2030	mpack_write_cstr(writer, cstr);
	2031	else
	2032	mpack_write_nil(writer);
	2033	}
	2034
	2035	void mpack_write_utf8(mpack_writer_t* writer, const char* str, uint32_t length) {
	2036	mpack_assert(str != NULL, "data for string of length %i is NULL", (int)length);
	2037	if (!mpack_utf8_check(str, length)) {
	2038	mpack_writer_flag_error(writer, mpack_error_invalid);
	2039	return;
	2040	}
	2041	mpack_write_str(writer, str, length);
	2042	}
	2043
	2044	void mpack_write_utf8_cstr(mpack_writer_t* writer, const char* cstr) {
	2045	mpack_assert(cstr != NULL, "cstr pointer is NULL");
	2046	size_t length = mpack_strlen(cstr);
	2047	if (length > UINT32_MAX) {
	2048	mpack_writer_flag_error(writer, mpack_error_invalid);
	2049	return;
	2050	}
	2051	mpack_write_utf8(writer, cstr, (uint32_t)length);
	2052	}
	2053
	2054	void mpack_write_utf8_cstr_or_nil(mpack_writer_t* writer, const char* cstr) {
	2055	if (cstr)
	2056	mpack_write_utf8_cstr(writer, cstr);
	2057	else
	2058	mpack_write_nil(writer);
	2059	}
	2060
	2061	#endif
	2062
	2063
	2064	/* mpack/mpack-reader.c.c */
	2065
	2066	#define MPACK_INTERNAL 1
	2067
	2068	/* #include "mpack-reader.h" */
	2069
	2070	#if MPACK_READER
	2071
	2072	static void mpack_reader_skip_using_fill(mpack_reader_t* reader, size_t count);
	2073
	2074	void mpack_reader_init(mpack_reader_t* reader, char* buffer, size_t size, size_t count) {
	2075	mpack_assert(buffer != NULL, "buffer is NULL");
	2076
	2077	mpack_memset(reader, 0, sizeof(*reader));
	2078	reader->buffer = buffer;
	2079	reader->size = size;
	2080	reader->data = buffer;
	2081	reader->end = buffer + count;
	2082
	2083	#if MPACK_READ_TRACKING
	2084	mpack_reader_flag_if_error(reader, mpack_track_init(&reader->track));
	2085	#endif
	2086
	2087	mpack_log("===========================\n");
	2088	mpack_log("initializing reader with buffer size %i\n", (int)size);
	2089	}
	2090
	2091	void mpack_reader_init_error(mpack_reader_t* reader, mpack_error_t error) {
	2092	mpack_memset(reader, 0, sizeof(*reader));
	2093	reader->error = error;
	2094
	2095	mpack_log("===========================\n");
	2096	mpack_log("initializing reader error state %i\n", (int)error);
	2097	}
	2098
	2099	void mpack_reader_init_data(mpack_reader_t* reader, const char* data, size_t count) {
	2100	mpack_assert(data != NULL, "data is NULL");
	2101
	2102	mpack_memset(reader, 0, sizeof(*reader));
	2103	reader->data = data;
	2104	reader->end = data + count;
	2105
	2106	#if MPACK_READ_TRACKING
	2107	mpack_reader_flag_if_error(reader, mpack_track_init(&reader->track));
	2108	#endif
	2109
	2110	mpack_log("===========================\n");
	2111	mpack_log("initializing reader with data size %i\n", (int)count);
	2112	}
	2113
	2114	void mpack_reader_set_fill(mpack_reader_t* reader, mpack_reader_fill_t fill) {
	2115	MPACK_STATIC_ASSERT(MPACK_READER_MINIMUM_BUFFER_SIZE >= MPACK_MAXIMUM_TAG_SIZE,
	2116	"minimum buffer size must fit any tag!");
	2117
	2118	if (reader->size == 0) {
	2119	mpack_break("cannot use fill function without a writeable buffer!");
	2120	mpack_reader_flag_error(reader, mpack_error_bug);
	2121	return;
	2122	}
	2123
	2124	if (reader->size < MPACK_READER_MINIMUM_BUFFER_SIZE) {
	2125	mpack_break("buffer size is %i, but minimum buffer size for fill is %i",
	2126	(int)reader->size, MPACK_READER_MINIMUM_BUFFER_SIZE);
	2127	mpack_reader_flag_error(reader, mpack_error_bug);
	2128	return;
	2129	}
	2130
	2131	reader->fill = fill;
	2132	}
	2133
	2134	void mpack_reader_set_skip(mpack_reader_t* reader, mpack_reader_skip_t skip) {
	2135	mpack_assert(reader->size != 0, "cannot use skip function without a writeable buffer!");
	2136	reader->skip = skip;
	2137	}
	2138
	2139	#if MPACK_STDIO
	2140	static size_t mpack_file_reader_fill(mpack_reader_t* reader, char* buffer, size_t count) {
	2141	if (feof((FILE *)reader->context)) {
	2142	mpack_reader_flag_error(reader, mpack_error_eof);
	2143	return 0;
	2144	}
	2145	return fread((void)buffer, 1, count, (FILE)reader->context);
	2146	}
	2147
	2148	static void mpack_file_reader_skip(mpack_reader_t* reader, size_t count) {
	2149	if (mpack_reader_error(reader) != mpack_ok)
	2150	return;
	2151	FILE* file = (FILE*)reader->context;
	2152
	2153	// We call ftell() to test whether the stream is seekable
	2154	// without causing a file error.
	2155	if (ftell(file) >= 0) {
	2156	mpack_log("seeking forward %i bytes\n", (int)count);
	2157	if (fseek(file, (long int)count, SEEK_CUR) == 0)
	2158	return;
	2159	mpack_log("fseek() didn't return zero!\n");
	2160	if (ferror(file)) {
	2161	mpack_reader_flag_error(reader, mpack_error_io);
	2162	return;
	2163	}
	2164	}
	2165
	2166	// If the stream is not seekable, fall back to the fill function.
	2167	mpack_reader_skip_using_fill(reader, count);
	2168	}
	2169
	2170	static void mpack_file_reader_teardown(mpack_reader_t* reader) {
	2171	MPACK_FREE(reader->buffer);
	2172	reader->buffer = NULL;
	2173	reader->context = NULL;
	2174	reader->size = 0;
	2175	reader->fill = NULL;
	2176	reader->skip = NULL;
	2177	reader->teardown = NULL;
	2178	}
	2179
	2180	static void mpack_file_reader_teardown_close(mpack_reader_t* reader) {
	2181	FILE* file = (FILE*)reader->context;
	2182
	2183	if (file) {
	2184	int ret = fclose(file);
	2185	if (ret != 0)
	2186	mpack_reader_flag_error(reader, mpack_error_io);
	2187	}
	2188
	2189	mpack_file_reader_teardown(reader);
	2190	}
	2191
	2192	void mpack_reader_init_stdfile(mpack_reader_t* reader, FILE* file, bool close_when_done) {
	2193	mpack_assert(file != NULL, "file is NULL");
	2194
	2195	size_t capacity = MPACK_BUFFER_SIZE;
	2196	char* buffer = (char*)MPACK_MALLOC(capacity);
	2197	if (buffer == NULL) {
	2198	mpack_reader_init_error(reader, mpack_error_memory);
	2199	if (close_when_done) {
	2200	fclose(file);
	2201	}
	2202	return;
	2203	}
	2204
	2205	mpack_reader_init(reader, buffer, capacity, 0);
	2206	mpack_reader_set_context(reader, file);
	2207	mpack_reader_set_fill(reader, mpack_file_reader_fill);
	2208	mpack_reader_set_skip(reader, mpack_file_reader_skip);
	2209	mpack_reader_set_teardown(reader, close_when_done ?
	2210	mpack_file_reader_teardown_close :
	2211	mpack_file_reader_teardown);
	2212	}
	2213
	2214	void mpack_reader_init_filename(mpack_reader_t* reader, const char* filename) {
	2215	mpack_assert(filename != NULL, "filename is NULL");
	2216
	2217	FILE* file = fopen(filename, "rb");
	2218	if (file == NULL) {
	2219	mpack_reader_init_error(reader, mpack_error_io);
	2220	return;
	2221	}
	2222
	2223	mpack_reader_init_stdfile(reader, file, true);
	2224	}
	2225	#endif
	2226
	2227	mpack_error_t mpack_reader_destroy(mpack_reader_t* reader) {
	2228
	2229	// clean up tracking, asserting if we're not already in an error state
	2230	#if MPACK_READ_TRACKING
	2231	mpack_reader_flag_if_error(reader, mpack_track_destroy(&reader->track, mpack_reader_error(reader) != mpack_ok));
	2232	#endif
	2233
	2234	if (reader->teardown)
	2235	reader->teardown(reader);
	2236	reader->teardown = NULL;
	2237
	2238	return reader->error;
	2239	}
	2240
	2241	size_t mpack_reader_remaining(mpack_reader_t* reader, const char** data) {
	2242	if (mpack_reader_error(reader) != mpack_ok)
	2243	return 0;
	2244
	2245	#if MPACK_READ_TRACKING
	2246	if (mpack_reader_flag_if_error(reader, mpack_track_check_empty(&reader->track)) != mpack_ok)
	2247	return 0;
	2248	#endif
	2249
	2250	if (data)
	2251	*data = reader->data;
	2252	return (size_t)(reader->end - reader->data);
	2253	}
	2254
	2255	void mpack_reader_flag_error(mpack_reader_t* reader, mpack_error_t error) {
	2256	mpack_log("reader %p setting error %i: %s\n", reader, (int)error, mpack_error_to_string(error));
	2257
	2258	if (reader->error == mpack_ok) {
	2259	reader->error = error;
	2260	reader->end = reader->data;
	2261	if (reader->error_fn)
	2262	reader->error_fn(reader, error);
	2263	}
	2264	}
	2265
	2266	// Loops on the fill function, reading between the minimum and
	2267	// maximum number of bytes and flagging an error if it fails.
	2268	MPACK_NOINLINE static size_t mpack_fill_range(mpack_reader_t* reader, char* p, size_t min_bytes, size_t max_bytes) {
	2269	mpack_assert(reader->fill != NULL, "mpack_fill_range() called with no fill function?");
	2270	mpack_assert(min_bytes > 0, "cannot fill zero bytes!");
	2271	mpack_assert(max_bytes >= min_bytes, "min_bytes %i cannot be larger than max_bytes %i!",
	2272	(int)min_bytes, (int)max_bytes);
	2273
	2274	size_t count = 0;
	2275	while (count < min_bytes) {
	2276	size_t read = reader->fill(reader, p + count, max_bytes - count);
	2277
	2278	// Reader fill functions can flag an error or return 0 on failure. We
	2279	// also guard against functions that -1 just in case.
	2280	if (mpack_reader_error(reader) != mpack_ok)
	2281	return 0;
	2282	if (read == 0 \|\| read == ((size_t)(-1))) {
	2283	mpack_reader_flag_error(reader, mpack_error_io);
	2284	return 0;
	2285	}
	2286
	2287	count += read;
	2288	}
	2289	return count;
	2290	}
	2291
	2292	MPACK_NOINLINE bool mpack_reader_ensure_straddle(mpack_reader_t* reader, size_t count) {
	2293	mpack_assert(count != 0, "cannot ensure zero bytes!");
	2294	mpack_assert(reader->error == mpack_ok, "reader cannot be in an error state!");
	2295
	2296	mpack_assert(count > (size_t)(reader->end - reader->data),
	2297	"straddling ensure requested for %i bytes, but there are %i bytes "
	2298	"left in buffer. call mpack_reader_ensure() instead",
	2299	(int)count, (int)(reader->end - reader->data));
	2300
	2301	// we'll need a fill function to get more data. if there's no
	2302	// fill function, the buffer should contain an entire MessagePack
	2303	// object, so we raise mpack_error_invalid instead of mpack_error_io
	2304	// on truncated data.
	2305	if (reader->fill == NULL) {
	2306	mpack_reader_flag_error(reader, mpack_error_invalid);
	2307	return false;
	2308	}
	2309
	2310	// we need enough space in the buffer. if the buffer is not
	2311	// big enough, we return mpack_error_too_big (since this is
	2312	// for an in-place read larger than the buffer size.)
	2313	if (count > reader->size) {
	2314	mpack_reader_flag_error(reader, mpack_error_too_big);
	2315	return false;
	2316	}
	2317
	2318	// move the existing data to the start of the buffer
	2319	size_t left = (size_t)(reader->end - reader->data);
	2320	mpack_memmove(reader->buffer, reader->data, left);
	2321	reader->end -= reader->data - reader->buffer;
	2322	reader->data = reader->buffer;
	2323
	2324	// read at least the necessary number of bytes, accepting up to the
	2325	// buffer size
	2326	size_t read = mpack_fill_range(reader, reader->buffer + left,
	2327	count - left, reader->size - left);
	2328	if (mpack_reader_error(reader) != mpack_ok)
	2329	return false;
	2330	reader->end += read;
	2331	return true;
	2332	}
	2333
	2334	// Reads count bytes into p. Used when there are not enough bytes
	2335	// left in the buffer to satisfy a read.
	2336	MPACK_NOINLINE void mpack_read_native_straddle(mpack_reader_t* reader, char* p, size_t count) {
	2337	mpack_assert(count == 0 \|\| p != NULL, "data pointer for %i bytes is NULL", (int)count);
	2338
	2339	if (mpack_reader_error(reader) != mpack_ok) {
	2340	mpack_memset(p, 0, count);
	2341	return;
	2342	}
	2343
	2344	size_t left = (size_t)(reader->end - reader->data);
	2345	mpack_log("big read for %i bytes into %p, %i left in buffer, buffer size %i\n",
	2346	(int)count, p, (int)left, (int)reader->size);
	2347
	2348	if (count <= left) {
	2349	mpack_assert(0,
	2350	"big read requested for %i bytes, but there are %i bytes "
	2351	"left in buffer. call mpack_read_native() instead",
	2352	(int)count, (int)left);
	2353	mpack_reader_flag_error(reader, mpack_error_bug);
	2354	mpack_memset(p, 0, count);
	2355	return;
	2356	}
	2357
	2358	// we'll need a fill function to get more data. if there's no
	2359	// fill function, the buffer should contain an entire MessagePack
	2360	// object, so we raise mpack_error_invalid instead of mpack_error_io
	2361	// on truncated data.
	2362	if (reader->fill == NULL) {
	2363	mpack_reader_flag_error(reader, mpack_error_invalid);
	2364	mpack_memset(p, 0, count);
	2365	return;
	2366	}
	2367
	2368	if (reader->size == 0) {
	2369	// somewhat debatable what error should be returned here. when
	2370	// initializing a reader with an in-memory buffer it's not
	2371	// necessarily a bug if the data is blank; it might just have
	2372	// been truncated to zero. for this reason we return the same
	2373	// error as if the data was truncated.
	2374	mpack_reader_flag_error(reader, mpack_error_io);
	2375	mpack_memset(p, 0, count);
	2376	return;
	2377	}
	2378
	2379	// flush what's left of the buffer
	2380	if (left > 0) {
	2381	mpack_log("flushing %i bytes remaining in buffer\n", (int)left);
	2382	mpack_memcpy(p, reader->data, left);
	2383	count -= left;
	2384	p += left;
	2385	reader->data += left;
	2386	}
	2387
	2388	// if the remaining data needed is some small fraction of the
	2389	// buffer size, we'll try to fill the buffer as much as possible
	2390	// and copy the needed data out.
	2391	if (count <= reader->size / MPACK_READER_SMALL_FRACTION_DENOMINATOR) {
	2392	size_t read = mpack_fill_range(reader, reader->buffer, count, reader->size);
	2393	if (mpack_reader_error(reader) != mpack_ok)
	2394	return;
	2395	mpack_memcpy(p, reader->buffer, count);
	2396	reader->data = reader->buffer + count;
	2397	reader->end = reader->buffer + read;
	2398
	2399	// otherwise we read the remaining data directly into the target.
	2400	} else {
	2401	mpack_log("reading %i additional bytes\n", (int)count);
	2402	mpack_fill_range(reader, p, count, count);
	2403	}
	2404	}
	2405
	2406	MPACK_NOINLINE static void mpack_skip_bytes_straddle(mpack_reader_t* reader, size_t count) {
	2407
	2408	// we'll need at least a fill function to skip more data. if there's
	2409	// no fill function, the buffer should contain an entire MessagePack
	2410	// object, so we raise mpack_error_invalid instead of mpack_error_io
	2411	// on truncated data. (see mpack_read_native_straddle())
	2412	if (reader->fill == NULL) {
	2413	mpack_log("reader has no fill function!\n");
	2414	mpack_reader_flag_error(reader, mpack_error_invalid);
	2415	return;
	2416	}
	2417
	2418	// discard whatever's left in the buffer
	2419	size_t left = (size_t)(reader->end - reader->data);
	2420	mpack_log("discarding %i bytes still in buffer\n", (int)left);
	2421	count -= left;
	2422	reader->data = reader->end;
	2423
	2424	// use the skip function if we've got one, and if we're trying
	2425	// to skip a lot of data. if we only need to skip some tiny
	2426	// fraction of the buffer size, it's probably better to just
	2427	// fill the buffer and skip from it instead of trying to seek.
	2428	if (reader->skip && count > reader->size / 16) {
	2429	mpack_log("calling skip function for %i bytes\n", (int)count);
	2430	reader->skip(reader, count);
	2431	return;
	2432	}
	2433
	2434	mpack_reader_skip_using_fill(reader, count);
	2435	}
	2436
	2437	void mpack_skip_bytes(mpack_reader_t* reader, size_t count) {
	2438	if (mpack_reader_error(reader) != mpack_ok)
	2439	return;
	2440	mpack_log("skip requested for %i bytes\n", (int)count);
	2441	mpack_reader_track_bytes(reader, count);
	2442
	2443	// check if we have enough in the buffer already
	2444	size_t left = (size_t)(reader->end - reader->data);
	2445	if (left >= count) {
	2446	mpack_log("skipping %i bytes still in buffer\n", (int)count);
	2447	reader->data += count;
	2448	return;
	2449	}
	2450
	2451	mpack_skip_bytes_straddle(reader, count);
	2452	}
	2453
	2454	MPACK_NOINLINE static void mpack_reader_skip_using_fill(mpack_reader_t* reader, size_t count) {
	2455	mpack_assert(reader->fill != NULL, "missing fill function!");
	2456	mpack_assert(reader->data == reader->end, "there are bytes left in the buffer!");
	2457	mpack_assert(reader->error == mpack_ok, "should not have called this in an error state (%i)", reader->error);
	2458	mpack_log("skip using fill for %i bytes\n", (int)count);
	2459
	2460	// fill and discard multiples of the buffer size
	2461	while (count > reader->size) {
	2462	mpack_log("filling and discarding buffer of %i bytes\n", (int)reader->size);
	2463	if (mpack_fill_range(reader, reader->buffer, reader->size, reader->size) < reader->size) {
	2464	mpack_reader_flag_error(reader, mpack_error_io);
	2465	return;
	2466	}
	2467	count -= reader->size;
	2468	}
	2469
	2470	// fill the buffer as much as possible
	2471	reader->data = reader->buffer;
	2472	size_t read = mpack_fill_range(reader, reader->buffer, count, reader->size);
	2473	if (read < count) {
	2474	mpack_reader_flag_error(reader, mpack_error_io);
	2475	return;
	2476	}
	2477	reader->end = reader->data + read;
	2478	mpack_log("filled %i bytes into buffer; discarding %i bytes\n", (int)read, (int)count);
	2479	reader->data += count;
	2480	}
	2481
	2482	void mpack_read_bytes(mpack_reader_t* reader, char* p, size_t count) {
	2483	mpack_assert(p != NULL, "destination for read of %i bytes is NULL", (int)count);
	2484	mpack_reader_track_bytes(reader, count);
	2485	mpack_read_native(reader, p, count);
	2486	}
	2487
	2488	void mpack_read_utf8(mpack_reader_t* reader, char* p, size_t byte_count) {
	2489	mpack_assert(p != NULL, "destination for read of %i bytes is NULL", (int)byte_count);
	2490	mpack_reader_track_str_bytes_all(reader, byte_count);
	2491	mpack_read_native(reader, p, byte_count);
	2492
	2493	if (mpack_reader_error(reader) == mpack_ok && !mpack_utf8_check(p, byte_count))
	2494	mpack_reader_flag_error(reader, mpack_error_type);
	2495	}
	2496
	2497	static void mpack_read_cstr_unchecked(mpack_reader_t* reader, char* buf, size_t buffer_size, size_t byte_count) {
	2498	mpack_assert(buf != NULL, "destination for read of %i bytes is NULL", (int)byte_count);
	2499	mpack_assert(buffer_size >= 1, "buffer size is zero; you must have room for at least a null-terminator");
	2500
	2501	if (mpack_reader_error(reader)) {
	2502	buf[0] = 0;
	2503	return;
	2504	}
	2505
	2506	if (byte_count > buffer_size - 1) {
	2507	mpack_reader_flag_error(reader, mpack_error_too_big);
	2508	buf[0] = 0;
	2509	return;
	2510	}
	2511
	2512	mpack_reader_track_str_bytes_all(reader, byte_count);
	2513	mpack_read_native(reader, buf, byte_count);
	2514	buf[byte_count] = 0;
	2515	}
	2516
	2517	void mpack_read_cstr(mpack_reader_t* reader, char* buf, size_t buffer_size, size_t byte_count) {
	2518	mpack_read_cstr_unchecked(reader, buf, buffer_size, byte_count);
	2519
	2520	// check for null bytes
	2521	if (mpack_reader_error(reader) == mpack_ok && !mpack_str_check_no_null(buf, byte_count)) {
	2522	buf[0] = 0;
	2523	mpack_reader_flag_error(reader, mpack_error_type);
	2524	}
	2525	}
	2526
	2527	void mpack_read_utf8_cstr(mpack_reader_t* reader, char* buf, size_t buffer_size, size_t byte_count) {
	2528	mpack_read_cstr_unchecked(reader, buf, buffer_size, byte_count);
	2529
	2530	// check encoding
	2531	if (mpack_reader_error(reader) == mpack_ok && !mpack_utf8_check_no_null(buf, byte_count)) {
	2532	buf[0] = 0;
	2533	mpack_reader_flag_error(reader, mpack_error_type);
	2534	}
	2535	}
	2536
	2537	#ifdef MPACK_MALLOC
	2538	// Reads native bytes with error callback disabled. This allows MPack reader functions
	2539	// to hold an allocated buffer and read native data into it without leaking it in
	2540	// case of a non-local jump (longjmp, throw) out of an error handler.
	2541	static void mpack_read_native_noerrorfn(mpack_reader_t* reader, char* p, size_t count) {
	2542	mpack_assert(reader->error == mpack_ok, "cannot call if an error is already flagged!");
	2543	mpack_reader_error_t error_fn = reader->error_fn;
	2544	reader->error_fn = NULL;
	2545	mpack_read_native(reader, p, count);
	2546	reader->error_fn = error_fn;
	2547	}
	2548
	2549	char* mpack_read_bytes_alloc_impl(mpack_reader_t* reader, size_t count, bool null_terminated) {
	2550
	2551	// track the bytes first in case it jumps
	2552	mpack_reader_track_bytes(reader, count);
	2553	if (mpack_reader_error(reader) != mpack_ok)
	2554	return NULL;
	2555
	2556	// cannot allocate zero bytes. this is not an error.
	2557	if (count == 0 && null_terminated == false)
	2558	return NULL;
	2559
	2560	// allocate data
	2561	char* data = (char*)MPACK_MALLOC(count + (null_terminated ? 1 : 0)); // TODO: can this overflow?
	2562	if (data == NULL) {
	2563	mpack_reader_flag_error(reader, mpack_error_memory);
	2564	return NULL;
	2565	}
	2566
	2567	// read with error callback disabled so we don't leak our buffer
	2568	mpack_read_native_noerrorfn(reader, data, count);
	2569
	2570	// report flagged errors
	2571	if (mpack_reader_error(reader) != mpack_ok) {
	2572	MPACK_FREE(data);
	2573	if (reader->error_fn)
	2574	reader->error_fn(reader, mpack_reader_error(reader));
	2575	return NULL;
	2576	}
	2577
	2578	if (null_terminated)
	2579	data[count] = '\0';
	2580	return data;
	2581	}
	2582	#endif
	2583
	2584	// read inplace without tracking (since there are different
	2585	// tracking modes for different inplace readers)
	2586	static const char* mpack_read_bytes_inplace_notrack(mpack_reader_t* reader, size_t count) {
	2587	if (mpack_reader_error(reader) != mpack_ok)
	2588	return NULL;
	2589
	2590	// if we have enough bytes already in the buffer, we can return it directly.
	2591	if ((size_t)(reader->end - reader->data) >= count) {
	2592	const char* bytes = reader->data;
	2593	reader->data += count;
	2594	return bytes;
	2595	}
	2596
	2597	if (!mpack_reader_ensure(reader, count))
	2598	return NULL;
	2599
	2600	const char* bytes = reader->data;
	2601	reader->data += count;
	2602	return bytes;
	2603	}
	2604
	2605	const char* mpack_read_bytes_inplace(mpack_reader_t* reader, size_t count) {
	2606	mpack_reader_track_bytes(reader, count);
	2607	return mpack_read_bytes_inplace_notrack(reader, count);
	2608	}
	2609
	2610	const char* mpack_read_utf8_inplace(mpack_reader_t* reader, size_t count) {
	2611	mpack_reader_track_str_bytes_all(reader, count);
	2612	const char* str = mpack_read_bytes_inplace_notrack(reader, count);
	2613
	2614	if (mpack_reader_error(reader) == mpack_ok && !mpack_utf8_check(str, count)) {
	2615	mpack_reader_flag_error(reader, mpack_error_type);
	2616	return NULL;
	2617	}
	2618
	2619	return str;
	2620	}
	2621
	2622	static size_t mpack_parse_tag(mpack_reader_t* reader, mpack_tag_t* tag) {
	2623	mpack_assert(reader->error == mpack_ok, "reader cannot be in an error state!");
	2624
	2625	if (!mpack_reader_ensure(reader, 1))
	2626	return 0;
	2627	uint8_t type = mpack_load_u8(reader->data);
	2628
	2629	// unfortunately, by far the fastest way to parse a tag is to switch
	2630	// on the first byte, and to explicitly list every possible byte. so for
	2631	// infix types, the list of cases is quite large.
	2632	//
	2633	// in size-optimized builds, we switch on the top four bits first to
	2634	// handle most infix types with a smaller jump table to save space.
	2635
	2636	#if MPACK_OPTIMIZE_FOR_SIZE
	2637	switch (type >> 4) {
	2638
	2639	// positive fixnum
	2640	case 0x0: case 0x1: case 0x2: case 0x3:
	2641	case 0x4: case 0x5: case 0x6: case 0x7:
	2642	*tag = mpack_tag_make_uint(type);
	2643	return 1;
	2644
	2645	// negative fixnum
	2646	case 0xe: case 0xf:
	2647	*tag = mpack_tag_make_int((int8_t)type);
	2648	return 1;
	2649
	2650	// fixmap
	2651	case 0x8:
	2652	*tag = mpack_tag_make_map(type & ~0xf0u);
	2653	return 1;
	2654
	2655	// fixarray
	2656	case 0x9:
	2657	*tag = mpack_tag_make_array(type & ~0xf0u);
	2658	return 1;
	2659
	2660	// fixstr
	2661	case 0xa: case 0xb:
	2662	*tag = mpack_tag_make_str(type & ~0xe0u);
	2663	return 1;
	2664
	2665	// not one of the common infix types
	2666	default:
	2667	break;
	2668
	2669	}
	2670	#endif
	2671
	2672	// handle individual type tags
	2673	switch (type) {
	2674
	2675	#if !MPACK_OPTIMIZE_FOR_SIZE
	2676	// positive fixnum
	2677	case 0x00: case 0x01: case 0x02: case 0x03: case 0x04: case 0x05: case 0x06: case 0x07:
	2678	case 0x08: case 0x09: case 0x0a: case 0x0b: case 0x0c: case 0x0d: case 0x0e: case 0x0f:
	2679	case 0x10: case 0x11: case 0x12: case 0x13: case 0x14: case 0x15: case 0x16: case 0x17:
	2680	case 0x18: case 0x19: case 0x1a: case 0x1b: case 0x1c: case 0x1d: case 0x1e: case 0x1f:
	2681	case 0x20: case 0x21: case 0x22: case 0x23: case 0x24: case 0x25: case 0x26: case 0x27:
	2682	case 0x28: case 0x29: case 0x2a: case 0x2b: case 0x2c: case 0x2d: case 0x2e: case 0x2f:
	2683	case 0x30: case 0x31: case 0x32: case 0x33: case 0x34: case 0x35: case 0x36: case 0x37:
	2684	case 0x38: case 0x39: case 0x3a: case 0x3b: case 0x3c: case 0x3d: case 0x3e: case 0x3f:
	2685	case 0x40: case 0x41: case 0x42: case 0x43: case 0x44: case 0x45: case 0x46: case 0x47:
	2686	case 0x48: case 0x49: case 0x4a: case 0x4b: case 0x4c: case 0x4d: case 0x4e: case 0x4f:
	2687	case 0x50: case 0x51: case 0x52: case 0x53: case 0x54: case 0x55: case 0x56: case 0x57:
	2688	case 0x58: case 0x59: case 0x5a: case 0x5b: case 0x5c: case 0x5d: case 0x5e: case 0x5f:
	2689	case 0x60: case 0x61: case 0x62: case 0x63: case 0x64: case 0x65: case 0x66: case 0x67:
	2690	case 0x68: case 0x69: case 0x6a: case 0x6b: case 0x6c: case 0x6d: case 0x6e: case 0x6f:
	2691	case 0x70: case 0x71: case 0x72: case 0x73: case 0x74: case 0x75: case 0x76: case 0x77:
	2692	case 0x78: case 0x79: case 0x7a: case 0x7b: case 0x7c: case 0x7d: case 0x7e: case 0x7f:
	2693	*tag = mpack_tag_make_uint(type);
	2694	return 1;
	2695
	2696	// negative fixnum
	2697	case 0xe0: case 0xe1: case 0xe2: case 0xe3: case 0xe4: case 0xe5: case 0xe6: case 0xe7:
	2698	case 0xe8: case 0xe9: case 0xea: case 0xeb: case 0xec: case 0xed: case 0xee: case 0xef:
	2699	case 0xf0: case 0xf1: case 0xf2: case 0xf3: case 0xf4: case 0xf5: case 0xf6: case 0xf7:
	2700	case 0xf8: case 0xf9: case 0xfa: case 0xfb: case 0xfc: case 0xfd: case 0xfe: case 0xff:
	2701	*tag = mpack_tag_make_int((int8_t)type);
	2702	return 1;
	2703
	2704	// fixmap
	2705	case 0x80: case 0x81: case 0x82: case 0x83: case 0x84: case 0x85: case 0x86: case 0x87:
	2706	case 0x88: case 0x89: case 0x8a: case 0x8b: case 0x8c: case 0x8d: case 0x8e: case 0x8f:
	2707	*tag = mpack_tag_make_map(type & ~0xf0u);
	2708	return 1;
	2709
	2710	// fixarray
	2711	case 0x90: case 0x91: case 0x92: case 0x93: case 0x94: case 0x95: case 0x96: case 0x97:
	2712	case 0x98: case 0x99: case 0x9a: case 0x9b: case 0x9c: case 0x9d: case 0x9e: case 0x9f:
	2713	*tag = mpack_tag_make_array(type & ~0xf0u);
	2714	return 1;
	2715
	2716	// fixstr
	2717	case 0xa0: case 0xa1: case 0xa2: case 0xa3: case 0xa4: case 0xa5: case 0xa6: case 0xa7:
	2718	case 0xa8: case 0xa9: case 0xaa: case 0xab: case 0xac: case 0xad: case 0xae: case 0xaf:
	2719	case 0xb0: case 0xb1: case 0xb2: case 0xb3: case 0xb4: case 0xb5: case 0xb6: case 0xb7:
	2720	case 0xb8: case 0xb9: case 0xba: case 0xbb: case 0xbc: case 0xbd: case 0xbe: case 0xbf:
	2721	*tag = mpack_tag_make_str(type & ~0xe0u);
	2722	return 1;
	2723	#endif
	2724
	2725	// nil
	2726	case 0xc0:
	2727	*tag = mpack_tag_make_nil();
	2728	return 1;
	2729
	2730	// bool
	2731	case 0xc2: case 0xc3:
	2732	*tag = mpack_tag_make_bool((bool)(type & 1));
	2733	return 1;
	2734
	2735	// bin8
	2736	case 0xc4:
	2737	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_BIN8))
	2738	return 0;
	2739	*tag = mpack_tag_make_bin(mpack_load_u8(reader->data + 1));
	2740	return MPACK_TAG_SIZE_BIN8;
	2741
	2742	// bin16
	2743	case 0xc5:
	2744	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_BIN16))
	2745	return 0;
	2746	*tag = mpack_tag_make_bin(mpack_load_u16(reader->data + 1));
	2747	return MPACK_TAG_SIZE_BIN16;
	2748
	2749	// bin32
	2750	case 0xc6:
	2751	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_BIN32))
	2752	return 0;
	2753	*tag = mpack_tag_make_bin(mpack_load_u32(reader->data + 1));
	2754	return MPACK_TAG_SIZE_BIN32;
	2755
	2756	#if MPACK_EXTENSIONS
	2757	// ext8
	2758	case 0xc7:
	2759	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_EXT8))
	2760	return 0;
	2761	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 2), mpack_load_u8(reader->data + 1));
	2762	return MPACK_TAG_SIZE_EXT8;
	2763
	2764	// ext16
	2765	case 0xc8:
	2766	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_EXT16))
	2767	return 0;
	2768	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 3), mpack_load_u16(reader->data + 1));
	2769	return MPACK_TAG_SIZE_EXT16;
	2770
	2771	// ext32
	2772	case 0xc9:
	2773	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_EXT32))
	2774	return 0;
	2775	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 5), mpack_load_u32(reader->data + 1));
	2776	return MPACK_TAG_SIZE_EXT32;
	2777	#endif
	2778
	2779	// float
	2780	case 0xca:
	2781	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FLOAT))
	2782	return 0;
	2783	*tag = mpack_tag_make_float(mpack_load_float(reader->data + 1));
	2784	return MPACK_TAG_SIZE_FLOAT;
	2785
	2786	// double
	2787	case 0xcb:
	2788	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_DOUBLE))
	2789	return 0;
	2790	*tag = mpack_tag_make_double(mpack_load_double(reader->data + 1));
	2791	return MPACK_TAG_SIZE_DOUBLE;
	2792
	2793	// uint8
	2794	case 0xcc:
	2795	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_U8))
	2796	return 0;
	2797	*tag = mpack_tag_make_uint(mpack_load_u8(reader->data + 1));
	2798	return MPACK_TAG_SIZE_U8;
	2799
	2800	// uint16
	2801	case 0xcd:
	2802	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_U16))
	2803	return 0;
	2804	*tag = mpack_tag_make_uint(mpack_load_u16(reader->data + 1));
	2805	return MPACK_TAG_SIZE_U16;
	2806
	2807	// uint32
	2808	case 0xce:
	2809	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_U32))
	2810	return 0;
	2811	*tag = mpack_tag_make_uint(mpack_load_u32(reader->data + 1));
	2812	return MPACK_TAG_SIZE_U32;
	2813
	2814	// uint64
	2815	case 0xcf:
	2816	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_U64))
	2817	return 0;
	2818	*tag = mpack_tag_make_uint(mpack_load_u64(reader->data + 1));
	2819	return MPACK_TAG_SIZE_U64;
	2820
	2821	// int8
	2822	case 0xd0:
	2823	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_I8))
	2824	return 0;
	2825	*tag = mpack_tag_make_int(mpack_load_i8(reader->data + 1));
	2826	return MPACK_TAG_SIZE_I8;
	2827
	2828	// int16
	2829	case 0xd1:
	2830	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_I16))
	2831	return 0;
	2832	*tag = mpack_tag_make_int(mpack_load_i16(reader->data + 1));
	2833	return MPACK_TAG_SIZE_I16;
	2834
	2835	// int32
	2836	case 0xd2:
	2837	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_I32))
	2838	return 0;
	2839	*tag = mpack_tag_make_int(mpack_load_i32(reader->data + 1));
	2840	return MPACK_TAG_SIZE_I32;
	2841
	2842	// int64
	2843	case 0xd3:
	2844	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_I64))
	2845	return 0;
	2846	*tag = mpack_tag_make_int(mpack_load_i64(reader->data + 1));
	2847	return MPACK_TAG_SIZE_I64;
	2848
	2849	#if MPACK_EXTENSIONS
	2850	// fixext1
	2851	case 0xd4:
	2852	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FIXEXT1))
	2853	return 0;
	2854	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 1), 1);
	2855	return MPACK_TAG_SIZE_FIXEXT1;
	2856
	2857	// fixext2
	2858	case 0xd5:
	2859	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FIXEXT2))
	2860	return 0;
	2861	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 1), 2);
	2862	return MPACK_TAG_SIZE_FIXEXT2;
	2863
	2864	// fixext4
	2865	case 0xd6:
	2866	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FIXEXT4))
	2867	return 0;
	2868	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 1), 4);
	2869	return 2;
	2870
	2871	// fixext8
	2872	case 0xd7:
	2873	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FIXEXT8))
	2874	return 0;
	2875	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 1), 8);
	2876	return MPACK_TAG_SIZE_FIXEXT8;
	2877
	2878	// fixext16
	2879	case 0xd8:
	2880	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_FIXEXT16))
	2881	return 0;
	2882	*tag = mpack_tag_make_ext(mpack_load_i8(reader->data + 1), 16);
	2883	return MPACK_TAG_SIZE_FIXEXT16;
	2884	#endif
	2885
	2886	// str8
	2887	case 0xd9:
	2888	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_STR8))
	2889	return 0;
	2890	*tag = mpack_tag_make_str(mpack_load_u8(reader->data + 1));
	2891	return MPACK_TAG_SIZE_STR8;
	2892
	2893	// str16
	2894	case 0xda:
	2895	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_STR16))
	2896	return 0;
	2897	*tag = mpack_tag_make_str(mpack_load_u16(reader->data + 1));
	2898	return MPACK_TAG_SIZE_STR16;
	2899
	2900	// str32
	2901	case 0xdb:
	2902	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_STR32))
	2903	return 0;
	2904	*tag = mpack_tag_make_str(mpack_load_u32(reader->data + 1));
	2905	return MPACK_TAG_SIZE_STR32;
	2906
	2907	// array16
	2908	case 0xdc:
	2909	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_ARRAY16))
	2910	return 0;
	2911	*tag = mpack_tag_make_array(mpack_load_u16(reader->data + 1));
	2912	return MPACK_TAG_SIZE_ARRAY16;
	2913
	2914	// array32
	2915	case 0xdd:
	2916	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_ARRAY32))
	2917	return 0;
	2918	*tag = mpack_tag_make_array(mpack_load_u32(reader->data + 1));
	2919	return MPACK_TAG_SIZE_ARRAY32;
	2920
	2921	// map16
	2922	case 0xde:
	2923	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_MAP16))
	2924	return 0;
	2925	*tag = mpack_tag_make_map(mpack_load_u16(reader->data + 1));
	2926	return MPACK_TAG_SIZE_MAP16;
	2927
	2928	// map32
	2929	case 0xdf:
	2930	if (!mpack_reader_ensure(reader, MPACK_TAG_SIZE_MAP32))
	2931	return 0;
	2932	*tag = mpack_tag_make_map(mpack_load_u32(reader->data + 1));
	2933	return MPACK_TAG_SIZE_MAP32;
	2934
	2935	// reserved
	2936	case 0xc1:
	2937	mpack_reader_flag_error(reader, mpack_error_invalid);
	2938	return 0;
	2939
	2940	#if !MPACK_EXTENSIONS
	2941	// ext
	2942	case 0xc7: // fallthrough
	2943	case 0xc8: // fallthrough
	2944	case 0xc9: // fallthrough
	2945	// fixext
	2946	case 0xd4: // fallthrough
	2947	case 0xd5: // fallthrough
	2948	case 0xd6: // fallthrough
	2949	case 0xd7: // fallthrough
	2950	case 0xd8:
	2951	mpack_reader_flag_error(reader, mpack_error_unsupported);
	2952	return 0;
	2953	#endif
	2954
	2955	#if MPACK_OPTIMIZE_FOR_SIZE
	2956	// any other bytes should have been handled by the infix switch
	2957	default:
	2958	break;
	2959	#endif
	2960	}
	2961
	2962	mpack_assert(0, "unreachable");
	2963	return 0;
	2964	}
	2965
	2966	mpack_tag_t mpack_read_tag(mpack_reader_t* reader) {
	2967	mpack_log("reading tag\n");
	2968
	2969	// make sure we can read a tag
	2970	if (mpack_reader_error(reader) != mpack_ok)
	2971	return mpack_tag_nil();
	2972	if (mpack_reader_track_element(reader) != mpack_ok)
	2973	return mpack_tag_nil();
	2974
	2975	mpack_tag_t tag = MPACK_TAG_ZERO;
	2976	size_t count = mpack_parse_tag(reader, &tag);
	2977	if (count == 0)
	2978	return mpack_tag_nil();
	2979
	2980	#if MPACK_READ_TRACKING
	2981	mpack_error_t track_error = mpack_ok;
	2982
	2983	switch (tag.type) {
	2984	case mpack_type_map:
	2985	case mpack_type_array:
	2986	track_error = mpack_track_push(&reader->track, tag.type, tag.v.n);
	2987	break;
	2988	#if MPACK_EXTENSIONS
	2989	case mpack_type_ext:
	2990	#endif
	2991	case mpack_type_str:
	2992	case mpack_type_bin:
	2993	track_error = mpack_track_push(&reader->track, tag.type, tag.v.l);
	2994	break;
	2995	default:
	2996	break;
	2997	}
	2998
	2999	if (track_error != mpack_ok) {
	3000	mpack_reader_flag_error(reader, track_error);
	3001	return mpack_tag_nil();
	3002	}
	3003	#endif
	3004
	3005	reader->data += count;
	3006	return tag;
	3007	}
	3008
	3009	mpack_tag_t mpack_peek_tag(mpack_reader_t* reader) {
	3010	mpack_log("peeking tag\n");
	3011
	3012	// make sure we can peek a tag
	3013	if (mpack_reader_error(reader) != mpack_ok)
	3014	return mpack_tag_nil();
	3015	if (mpack_reader_track_peek_element(reader) != mpack_ok)
	3016	return mpack_tag_nil();
	3017
	3018	mpack_tag_t tag = MPACK_TAG_ZERO;
	3019	if (mpack_parse_tag(reader, &tag) == 0)
	3020	return mpack_tag_nil();
	3021	return tag;
	3022	}
	3023
	3024	void mpack_discard(mpack_reader_t* reader) {
	3025	mpack_tag_t var = mpack_read_tag(reader);
	3026	if (mpack_reader_error(reader))
	3027	return;
	3028	switch (var.type) {
	3029	case mpack_type_str:
	3030	mpack_skip_bytes(reader, var.v.l);
	3031	mpack_done_str(reader);
	3032	break;
	3033	case mpack_type_bin:
	3034	mpack_skip_bytes(reader, var.v.l);
	3035	mpack_done_bin(reader);
	3036	break;
	3037	#if MPACK_EXTENSIONS
	3038	case mpack_type_ext:
	3039	mpack_skip_bytes(reader, var.v.l);
	3040	mpack_done_ext(reader);
	3041	break;
	3042	#endif
	3043	case mpack_type_array: {
	3044	for (; var.v.n > 0; --var.v.n) {
	3045	mpack_discard(reader);
	3046	if (mpack_reader_error(reader))
	3047	break;
	3048	}
	3049	mpack_done_array(reader);
	3050	break;
	3051	}
	3052	case mpack_type_map: {
	3053	for (; var.v.n > 0; --var.v.n) {
	3054	mpack_discard(reader);
	3055	mpack_discard(reader);
	3056	if (mpack_reader_error(reader))
	3057	break;
	3058	}
	3059	mpack_done_map(reader);
	3060	break;
	3061	}
	3062	default:
	3063	break;
	3064	}
	3065	}
	3066
	3067	#if MPACK_EXTENSIONS
	3068	mpack_timestamp_t mpack_read_timestamp(mpack_reader_t* reader, size_t size) {
	3069	mpack_timestamp_t timestamp = {0, 0};
	3070
	3071	if (size != 4 && size != 8 && size != 12) {
	3072	mpack_reader_flag_error(reader, mpack_error_invalid);
	3073	return timestamp;
	3074	}
	3075
	3076	char buf[12];
	3077	mpack_read_bytes(reader, buf, size);
	3078	mpack_done_ext(reader);
	3079	if (mpack_reader_error(reader) != mpack_ok)
	3080	return timestamp;
	3081
	3082	switch (size) {
	3083	case 4:
	3084	timestamp.seconds = (int64_t)(uint64_t)mpack_load_u32(buf);
	3085	break;
	3086
	3087	case 8: {
	3088	uint64_t packed = mpack_load_u64(buf);
	3089	timestamp.seconds = (int64_t)(packed & ((UINT64_C(1) << 34) - 1));
	3090	timestamp.nanoseconds = (uint32_t)(packed >> 34);
	3091	break;
	3092	}
	3093
	3094	case 12:
	3095	timestamp.nanoseconds = mpack_load_u32(buf);
	3096	timestamp.seconds = mpack_load_i64(buf + 4);
	3097	break;
	3098
	3099	default:
	3100	mpack_assert(false, "unreachable");
	3101	break;
	3102	}
	3103
	3104	if (timestamp.nanoseconds > MPACK_TIMESTAMP_NANOSECONDS_MAX) {
	3105	mpack_reader_flag_error(reader, mpack_error_invalid);
	3106	mpack_timestamp_t zero = {0, 0};
	3107	return zero;
	3108	}
	3109
	3110	return timestamp;
	3111	}
	3112	#endif
	3113
	3114	#if MPACK_READ_TRACKING
	3115	void mpack_done_type(mpack_reader_t* reader, mpack_type_t type) {
	3116	if (mpack_reader_error(reader) == mpack_ok)
	3117	mpack_reader_flag_if_error(reader, mpack_track_pop(&reader->track, type));
	3118	}
	3119	#endif
	3120
	3121	#if MPACK_DEBUG && MPACK_STDIO
	3122	static size_t mpack_print_read_prefix(mpack_reader_t* reader, size_t length, char* buffer, size_t buffer_size) {
	3123	if (length == 0)
	3124	return 0;
	3125
	3126	size_t read = (length < buffer_size) ? length : buffer_size;
	3127	mpack_read_bytes(reader, buffer, read);
	3128	if (mpack_reader_error(reader) != mpack_ok)
	3129	return 0;
	3130
	3131	mpack_skip_bytes(reader, length - read);
	3132	return read;
	3133	}
	3134
	3135	static void mpack_print_element(mpack_reader_t* reader, mpack_print_t* print, size_t depth) {
	3136	mpack_tag_t val = mpack_read_tag(reader);
	3137	if (mpack_reader_error(reader) != mpack_ok)
	3138	return;
	3139
	3140	// We read some bytes from bin and ext so we can print its prefix in hex.
	3141	char buffer[MPACK_PRINT_BYTE_COUNT];
	3142	size_t count = 0;
	3143
	3144	switch (val.type) {
	3145	case mpack_type_str:
	3146	mpack_print_append_cstr(print, "\"");
	3147	for (size_t i = 0; i < val.v.l; ++i) {
	3148	char c;
	3149	mpack_read_bytes(reader, &c, 1);
	3150	if (mpack_reader_error(reader) != mpack_ok)
	3151	return;
	3152	switch (c) {
	3153	case '\n': mpack_print_append_cstr(print, "\\n"); break;
	3154	case '\\': mpack_print_append_cstr(print, "\\\\"); break;
	3155	case '"': mpack_print_append_cstr(print, "\\\""); break;
	3156	default: mpack_print_append(print, &c, 1); break;
	3157	}
	3158	}
	3159	mpack_print_append_cstr(print, "\"");
	3160	mpack_done_str(reader);
	3161	return;
	3162
	3163	case mpack_type_array:
	3164	mpack_print_append_cstr(print, "[\n");
	3165	for (size_t i = 0; i < val.v.n; ++i) {
	3166	for (size_t j = 0; j < depth + 1; ++j)
	3167	mpack_print_append_cstr(print, " ");
	3168	mpack_print_element(reader, print, depth + 1);
	3169	if (mpack_reader_error(reader) != mpack_ok)
	3170	return;
	3171	if (i != val.v.n - 1)
	3172	mpack_print_append_cstr(print, ",");
	3173	mpack_print_append_cstr(print, "\n");
	3174	}
	3175	for (size_t i = 0; i < depth; ++i)
	3176	mpack_print_append_cstr(print, " ");
	3177	mpack_print_append_cstr(print, "]");
	3178	mpack_done_array(reader);
	3179	return;
	3180
	3181	case mpack_type_map:
	3182	mpack_print_append_cstr(print, "{\n");
	3183	for (size_t i = 0; i < val.v.n; ++i) {
	3184	for (size_t j = 0; j < depth + 1; ++j)
	3185	mpack_print_append_cstr(print, " ");
	3186	mpack_print_element(reader, print, depth + 1);
	3187	if (mpack_reader_error(reader) != mpack_ok)
	3188	return;
	3189	mpack_print_append_cstr(print, ": ");
	3190	mpack_print_element(reader, print, depth + 1);
	3191	if (mpack_reader_error(reader) != mpack_ok)
	3192	return;
	3193	if (i != val.v.n - 1)
	3194	mpack_print_append_cstr(print, ",");
	3195	mpack_print_append_cstr(print, "\n");
	3196	}
	3197	for (size_t i = 0; i < depth; ++i)
	3198	mpack_print_append_cstr(print, " ");
	3199	mpack_print_append_cstr(print, "}");
	3200	mpack_done_map(reader);
	3201	return;
	3202
	3203	// The above cases return so as not to print a pseudo-json value. The
	3204	// below cases break and print pseudo-json.
	3205
	3206	case mpack_type_bin:
	3207	count = mpack_print_read_prefix(reader, mpack_tag_bin_length(&val), buffer, sizeof(buffer));
	3208	mpack_done_bin(reader);
	3209	break;
	3210
	3211	#if MPACK_EXTENSIONS
	3212	case mpack_type_ext:
	3213	count = mpack_print_read_prefix(reader, mpack_tag_ext_length(&val), buffer, sizeof(buffer));
	3214	mpack_done_ext(reader);
	3215	break;
	3216	#endif
	3217
	3218	default:
	3219	break;
	3220	}
	3221
	3222	char buf[256];
	3223	mpack_tag_debug_pseudo_json(val, buf, sizeof(buf), buffer, count);
	3224	mpack_print_append_cstr(print, buf);
	3225	}
	3226
	3227	static void mpack_print_and_destroy(mpack_reader_t* reader, mpack_print_t* print, size_t depth) {
	3228	for (size_t i = 0; i < depth; ++i)
	3229	mpack_print_append_cstr(print, " ");
	3230	mpack_print_element(reader, print, depth);
	3231
	3232	size_t remaining = mpack_reader_remaining(reader, NULL);
	3233
	3234	char buf[256];
	3235	if (mpack_reader_destroy(reader) != mpack_ok) {
	3236	mpack_snprintf(buf, sizeof(buf), "\n<mpack parsing error %s>", mpack_error_to_string(mpack_reader_error(reader)));
	3237	buf[sizeof(buf) - 1] = '\0';
	3238	mpack_print_append_cstr(print, buf);
	3239	} else if (remaining > 0) {
	3240	mpack_snprintf(buf, sizeof(buf), "\n<%i extra bytes at end of message>", (int)remaining);
	3241	buf[sizeof(buf) - 1] = '\0';
	3242	mpack_print_append_cstr(print, buf);
	3243	}
	3244	}
	3245
	3246	static void mpack_print_data(const char* data, size_t len, mpack_print_t* print, size_t depth) {
	3247	mpack_reader_t reader;
	3248	mpack_reader_init_data(&reader, data, len);
	3249	mpack_print_and_destroy(&reader, print, depth);
	3250	}
	3251
	3252	void mpack_print_data_to_buffer(const char* data, size_t data_size, char* buffer, size_t buffer_size) {
	3253	if (buffer_size == 0) {
	3254	mpack_assert(false, "buffer size is zero!");
	3255	return;
	3256	}
	3257
	3258	mpack_print_t print;
	3259	mpack_memset(&print, 0, sizeof(print));
	3260	print.buffer = buffer;
	3261	print.size = buffer_size;
	3262	mpack_print_data(data, data_size, &print, 0);
	3263	mpack_print_append(&print, "", 1); // null-terminator
	3264	mpack_print_flush(&print);
	3265
	3266	// we always make sure there's a null-terminator at the end of the buffer
	3267	// in case we ran out of space.
	3268	print.buffer[print.size - 1] = '\0';
	3269	}
	3270
	3271	void mpack_print_data_to_callback(const char* data, size_t size, mpack_print_callback_t callback, void* context) {
	3272	char buffer[1024];
	3273	mpack_print_t print;
	3274	mpack_memset(&print, 0, sizeof(print));
	3275	print.buffer = buffer;
	3276	print.size = sizeof(buffer);
	3277	print.callback = callback;
	3278	print.context = context;
	3279	mpack_print_data(data, size, &print, 0);
	3280	mpack_print_flush(&print);
	3281	}
	3282
	3283	void mpack_print_data_to_file(const char* data, size_t len, FILE* file) {
	3284	mpack_assert(data != NULL, "data is NULL");
	3285	mpack_assert(file != NULL, "file is NULL");
	3286
	3287	char buffer[1024];
	3288	mpack_print_t print;
	3289	mpack_memset(&print, 0, sizeof(print));
	3290	print.buffer = buffer;
	3291	print.size = sizeof(buffer);
	3292	print.callback = &mpack_print_file_callback;
	3293	print.context = file;
	3294
	3295	mpack_print_data(data, len, &print, 2);
	3296	mpack_print_append_cstr(&print, "\n");
	3297	mpack_print_flush(&print);
	3298	}
	3299
	3300	void mpack_print_stdfile_to_callback(FILE* file, mpack_print_callback_t callback, void* context) {
	3301	char buffer[1024];
	3302	mpack_print_t print;
	3303	mpack_memset(&print, 0, sizeof(print));
	3304	print.buffer = buffer;
	3305	print.size = sizeof(buffer);
	3306	print.callback = callback;
	3307	print.context = context;
	3308
	3309	mpack_reader_t reader;
	3310	mpack_reader_init_stdfile(&reader, file, false);
	3311	mpack_print_and_destroy(&reader, &print, 0);
	3312	mpack_print_flush(&print);
	3313	}
	3314	#endif
	3315
	3316	#endif
	3317
	3318	/* mpack/mpack-expect.c.c */
	3319
	3320	#define MPACK_INTERNAL 1
	3321
	3322	/* #include "mpack-expect.h" */
	3323
	3324	#if MPACK_EXPECT
	3325
	3326
	3327	// Helpers
	3328
	3329	MPACK_STATIC_INLINE uint8_t mpack_expect_native_u8(mpack_reader_t* reader) {
	3330	if (mpack_reader_error(reader) != mpack_ok)
	3331	return 0;
	3332	uint8_t type;
	3333	if (!mpack_reader_ensure(reader, sizeof(type)))
	3334	return 0;
	3335	type = mpack_load_u8(reader->data);
	3336	reader->data += sizeof(type);
	3337	return type;
	3338	}
	3339
	3340	#if !MPACK_OPTIMIZE_FOR_SIZE
	3341	MPACK_STATIC_INLINE uint16_t mpack_expect_native_u16(mpack_reader_t* reader) {
	3342	if (mpack_reader_error(reader) != mpack_ok)
	3343	return 0;
	3344	uint16_t type;
	3345	if (!mpack_reader_ensure(reader, sizeof(type)))
	3346	return 0;
	3347	type = mpack_load_u16(reader->data);
	3348	reader->data += sizeof(type);
	3349	return type;
	3350	}
	3351
	3352	MPACK_STATIC_INLINE uint32_t mpack_expect_native_u32(mpack_reader_t* reader) {
	3353	if (mpack_reader_error(reader) != mpack_ok)
	3354	return 0;
	3355	uint32_t type;
	3356	if (!mpack_reader_ensure(reader, sizeof(type)))
	3357	return 0;
	3358	type = mpack_load_u32(reader->data);
	3359	reader->data += sizeof(type);
	3360	return type;
	3361	}
	3362	#endif
	3363
	3364	MPACK_STATIC_INLINE uint8_t mpack_expect_type_byte(mpack_reader_t* reader) {
	3365	mpack_reader_track_element(reader);
	3366	return mpack_expect_native_u8(reader);
	3367	}
	3368
	3369
	3370	// Basic Number Functions
	3371
	3372	uint8_t mpack_expect_u8(mpack_reader_t* reader) {
	3373	mpack_tag_t var = mpack_read_tag(reader);
	3374	if (var.type == mpack_type_uint) {
	3375	if (var.v.u <= UINT8_MAX)
	3376	return (uint8_t)var.v.u;
	3377	} else if (var.type == mpack_type_int) {
	3378	if (var.v.i >= 0 && var.v.i <= UINT8_MAX)
	3379	return (uint8_t)var.v.i;
	3380	}
	3381	mpack_reader_flag_error(reader, mpack_error_type);
	3382	return 0;
	3383	}
	3384
	3385	uint16_t mpack_expect_u16(mpack_reader_t* reader) {
	3386	mpack_tag_t var = mpack_read_tag(reader);
	3387	if (var.type == mpack_type_uint) {
	3388	if (var.v.u <= UINT16_MAX)
	3389	return (uint16_t)var.v.u;
	3390	} else if (var.type == mpack_type_int) {
	3391	if (var.v.i >= 0 && var.v.i <= UINT16_MAX)
	3392	return (uint16_t)var.v.i;
	3393	}
	3394	mpack_reader_flag_error(reader, mpack_error_type);
	3395	return 0;
	3396	}
	3397
	3398	uint32_t mpack_expect_u32(mpack_reader_t* reader) {
	3399	mpack_tag_t var = mpack_read_tag(reader);
	3400	if (var.type == mpack_type_uint) {
	3401	if (var.v.u <= UINT32_MAX)
	3402	return (uint32_t)var.v.u;
	3403	} else if (var.type == mpack_type_int) {
	3404	if (var.v.i >= 0 && var.v.i <= UINT32_MAX)
	3405	return (uint32_t)var.v.i;
	3406	}
	3407	mpack_reader_flag_error(reader, mpack_error_type);
	3408	return 0;
	3409	}
	3410
	3411	uint64_t mpack_expect_u64(mpack_reader_t* reader) {
	3412	mpack_tag_t var = mpack_read_tag(reader);
	3413	if (var.type == mpack_type_uint) {
	3414	return var.v.u;
	3415	} else if (var.type == mpack_type_int) {
	3416	if (var.v.i >= 0)
	3417	return (uint64_t)var.v.i;
	3418	}
	3419	mpack_reader_flag_error(reader, mpack_error_type);
	3420	return 0;
	3421	}
	3422
	3423	int8_t mpack_expect_i8(mpack_reader_t* reader) {
	3424	mpack_tag_t var = mpack_read_tag(reader);
	3425	if (var.type == mpack_type_uint) {
	3426	if (var.v.u <= INT8_MAX)
	3427	return (int8_t)var.v.u;
	3428	} else if (var.type == mpack_type_int) {
	3429	if (var.v.i >= INT8_MIN && var.v.i <= INT8_MAX)
	3430	return (int8_t)var.v.i;
	3431	}
	3432	mpack_reader_flag_error(reader, mpack_error_type);
	3433	return 0;
	3434	}
	3435
	3436	int16_t mpack_expect_i16(mpack_reader_t* reader) {
	3437	mpack_tag_t var = mpack_read_tag(reader);
	3438	if (var.type == mpack_type_uint) {
	3439	if (var.v.u <= INT16_MAX)
	3440	return (int16_t)var.v.u;
	3441	} else if (var.type == mpack_type_int) {
	3442	if (var.v.i >= INT16_MIN && var.v.i <= INT16_MAX)
	3443	return (int16_t)var.v.i;
	3444	}
	3445	mpack_reader_flag_error(reader, mpack_error_type);
	3446	return 0;
	3447	}
	3448
	3449	int32_t mpack_expect_i32(mpack_reader_t* reader) {
	3450	mpack_tag_t var = mpack_read_tag(reader);
	3451	if (var.type == mpack_type_uint) {
	3452	if (var.v.u <= INT32_MAX)
	3453	return (int32_t)var.v.u;
	3454	} else if (var.type == mpack_type_int) {
	3455	if (var.v.i >= INT32_MIN && var.v.i <= INT32_MAX)
	3456	return (int32_t)var.v.i;
	3457	}
	3458	mpack_reader_flag_error(reader, mpack_error_type);
	3459	return 0;
	3460	}
	3461
	3462	int64_t mpack_expect_i64(mpack_reader_t* reader) {
	3463	mpack_tag_t var = mpack_read_tag(reader);
	3464	if (var.type == mpack_type_uint) {
	3465	if (var.v.u <= INT64_MAX)
	3466	return (int64_t)var.v.u;
	3467	} else if (var.type == mpack_type_int) {
	3468	return var.v.i;
	3469	}
	3470	mpack_reader_flag_error(reader, mpack_error_type);
	3471	return 0;
	3472	}
	3473
	3474	float mpack_expect_float(mpack_reader_t* reader) {
	3475	mpack_tag_t var = mpack_read_tag(reader);
	3476	if (var.type == mpack_type_uint)
	3477	return (float)var.v.u;
	3478	else if (var.type == mpack_type_int)
	3479	return (float)var.v.i;
	3480	else if (var.type == mpack_type_float)
	3481	return var.v.f;
	3482	else if (var.type == mpack_type_double)
	3483	return (float)var.v.d;
	3484	mpack_reader_flag_error(reader, mpack_error_type);
	3485	return 0.0f;
	3486	}
	3487
	3488	double mpack_expect_double(mpack_reader_t* reader) {
	3489	mpack_tag_t var = mpack_read_tag(reader);
	3490	if (var.type == mpack_type_uint)
	3491	return (double)var.v.u;
	3492	else if (var.type == mpack_type_int)
	3493	return (double)var.v.i;
	3494	else if (var.type == mpack_type_float)
	3495	return (double)var.v.f;
	3496	else if (var.type == mpack_type_double)
	3497	return var.v.d;
	3498	mpack_reader_flag_error(reader, mpack_error_type);
	3499	return 0.0;
	3500	}
	3501
	3502	float mpack_expect_float_strict(mpack_reader_t* reader) {
	3503	mpack_tag_t var = mpack_read_tag(reader);
	3504	if (var.type == mpack_type_float)
	3505	return var.v.f;
	3506	mpack_reader_flag_error(reader, mpack_error_type);
	3507	return 0.0f;
	3508	}
	3509
	3510	double mpack_expect_double_strict(mpack_reader_t* reader) {
	3511	mpack_tag_t var = mpack_read_tag(reader);
	3512	if (var.type == mpack_type_float)
	3513	return (double)var.v.f;
	3514	else if (var.type == mpack_type_double)
	3515	return var.v.d;
	3516	mpack_reader_flag_error(reader, mpack_error_type);
	3517	return 0.0;
	3518	}
	3519
	3520
	3521	// Ranged Number Functions
	3522	//
	3523	// All ranged functions are identical other than the type, so we
	3524	// define their content with a macro. The prototypes are still written
	3525	// out in full to support ctags/IDE tools.
	3526
	3527	#define MPACK_EXPECT_RANGE_IMPL(name, type_t) \
	3528	\
	3529	/* make sure the range is sensible */ \
	3530	mpack_assert(min_value <= max_value, \
	3531	"min_value %i must be less than or equal to max_value %i", \
	3532	min_value, max_value); \
	3533	\
	3534	/* read the value */ \
	3535	type_t val = mpack_expect_##name(reader); \
	3536	if (mpack_reader_error(reader) != mpack_ok) \
	3537	return min_value; \
	3538	\
	3539	/* make sure it fits */ \
	3540	if (val < min_value \|\| val > max_value) { \
	3541	mpack_reader_flag_error(reader, mpack_error_type); \
	3542	return min_value; \
	3543	} \
	3544	\
	3545	return val;
	3546
	3547	uint8_t mpack_expect_u8_range(mpack_reader_t* reader, uint8_t min_value, uint8_t max_value) {MPACK_EXPECT_RANGE_IMPL(u8, uint8_t)}
	3548	uint16_t mpack_expect_u16_range(mpack_reader_t* reader, uint16_t min_value, uint16_t max_value) {MPACK_EXPECT_RANGE_IMPL(u16, uint16_t)}
	3549	uint32_t mpack_expect_u32_range(mpack_reader_t* reader, uint32_t min_value, uint32_t max_value) {MPACK_EXPECT_RANGE_IMPL(u32, uint32_t)}
	3550	uint64_t mpack_expect_u64_range(mpack_reader_t* reader, uint64_t min_value, uint64_t max_value) {MPACK_EXPECT_RANGE_IMPL(u64, uint64_t)}
	3551
	3552	int8_t mpack_expect_i8_range(mpack_reader_t* reader, int8_t min_value, int8_t max_value) {MPACK_EXPECT_RANGE_IMPL(i8, int8_t)}
	3553	int16_t mpack_expect_i16_range(mpack_reader_t* reader, int16_t min_value, int16_t max_value) {MPACK_EXPECT_RANGE_IMPL(i16, int16_t)}
	3554	int32_t mpack_expect_i32_range(mpack_reader_t* reader, int32_t min_value, int32_t max_value) {MPACK_EXPECT_RANGE_IMPL(i32, int32_t)}
	3555	int64_t mpack_expect_i64_range(mpack_reader_t* reader, int64_t min_value, int64_t max_value) {MPACK_EXPECT_RANGE_IMPL(i64, int64_t)}
	3556
	3557	float mpack_expect_float_range(mpack_reader_t* reader, float min_value, float max_value) {MPACK_EXPECT_RANGE_IMPL(float, float)}
	3558	double mpack_expect_double_range(mpack_reader_t* reader, double min_value, double max_value) {MPACK_EXPECT_RANGE_IMPL(double, double)}
	3559
	3560	uint32_t mpack_expect_map_range(mpack_reader_t* reader, uint32_t min_value, uint32_t max_value) {MPACK_EXPECT_RANGE_IMPL(map, uint32_t)}
	3561	uint32_t mpack_expect_array_range(mpack_reader_t* reader, uint32_t min_value, uint32_t max_value) {MPACK_EXPECT_RANGE_IMPL(array, uint32_t)}
	3562
	3563
	3564	// Matching Number Functions
	3565
	3566	void mpack_expect_uint_match(mpack_reader_t* reader, uint64_t value) {
	3567	if (mpack_expect_u64(reader) != value)
	3568	mpack_reader_flag_error(reader, mpack_error_type);
	3569	}
	3570
	3571	void mpack_expect_int_match(mpack_reader_t* reader, int64_t value) {
	3572	if (mpack_expect_i64(reader) != value)
	3573	mpack_reader_flag_error(reader, mpack_error_type);
	3574	}
	3575
	3576
	3577	// Other Basic Types
	3578
	3579	void mpack_expect_nil(mpack_reader_t* reader) {
	3580	if (mpack_expect_type_byte(reader) != 0xc0)
	3581	mpack_reader_flag_error(reader, mpack_error_type);
	3582	}
	3583
	3584	bool mpack_expect_bool(mpack_reader_t* reader) {
	3585	uint8_t type = mpack_expect_type_byte(reader);
	3586	if ((type & ~1) != 0xc2)
	3587	mpack_reader_flag_error(reader, mpack_error_type);
	3588	return (bool)(type & 1);
	3589	}
	3590
	3591	void mpack_expect_true(mpack_reader_t* reader) {
	3592	if (mpack_expect_bool(reader) != true)
	3593	mpack_reader_flag_error(reader, mpack_error_type);
	3594	}
	3595
	3596	void mpack_expect_false(mpack_reader_t* reader) {
	3597	if (mpack_expect_bool(reader) != false)
	3598	mpack_reader_flag_error(reader, mpack_error_type);
	3599	}
	3600
	3601	#if MPACK_EXTENSIONS
	3602	mpack_timestamp_t mpack_expect_timestamp(mpack_reader_t* reader) {
	3603	mpack_timestamp_t zero = {0, 0};
	3604
	3605	mpack_tag_t tag = mpack_read_tag(reader);
	3606	if (tag.type != mpack_type_ext) {
	3607	mpack_reader_flag_error(reader, mpack_error_type);
	3608	return zero;
	3609	}
	3610	if (mpack_tag_ext_exttype(&tag) != MPACK_EXTTYPE_TIMESTAMP) {
	3611	mpack_reader_flag_error(reader, mpack_error_type);
	3612	return zero;
	3613	}
	3614
	3615	return mpack_read_timestamp(reader, mpack_tag_ext_length(&tag));
	3616	}
	3617
	3618	int64_t mpack_expect_timestamp_truncate(mpack_reader_t* reader) {
	3619	return mpack_expect_timestamp(reader).seconds;
	3620	}
	3621	#endif
	3622
	3623
	3624	// Compound Types
	3625
	3626	uint32_t mpack_expect_map(mpack_reader_t* reader) {
	3627	mpack_tag_t var = mpack_read_tag(reader);
	3628	if (var.type == mpack_type_map)
	3629	return var.v.n;
	3630	mpack_reader_flag_error(reader, mpack_error_type);
	3631	return 0;
	3632	}
	3633
	3634	void mpack_expect_map_match(mpack_reader_t* reader, uint32_t count) {
	3635	if (mpack_expect_map(reader) != count)
	3636	mpack_reader_flag_error(reader, mpack_error_type);
	3637	}
	3638
	3639	bool mpack_expect_map_or_nil(mpack_reader_t* reader, uint32_t* count) {
	3640	mpack_assert(count != NULL, "count cannot be NULL");
	3641
	3642	mpack_tag_t var = mpack_read_tag(reader);
	3643	if (var.type == mpack_type_nil) {
	3644	*count = 0;
	3645	return false;
	3646	}
	3647	if (var.type == mpack_type_map) {
	3648	*count = var.v.n;
	3649	return true;
	3650	}
	3651	mpack_reader_flag_error(reader, mpack_error_type);
	3652	*count = 0;
	3653	return false;
	3654	}
	3655
	3656	bool mpack_expect_map_max_or_nil(mpack_reader_t* reader, uint32_t max_count, uint32_t* count) {
	3657	mpack_assert(count != NULL, "count cannot be NULL");
	3658
	3659	bool has_map = mpack_expect_map_or_nil(reader, count);
	3660	if (has_map && *count > max_count) {
	3661	*count = 0;
	3662	mpack_reader_flag_error(reader, mpack_error_type);
	3663	return false;
	3664	}
	3665	return has_map;
	3666	}
	3667
	3668	uint32_t mpack_expect_array(mpack_reader_t* reader) {
	3669	mpack_tag_t var = mpack_read_tag(reader);
	3670	if (var.type == mpack_type_array)
	3671	return var.v.n;
	3672	mpack_reader_flag_error(reader, mpack_error_type);
	3673	return 0;
	3674	}
	3675
	3676	void mpack_expect_array_match(mpack_reader_t* reader, uint32_t count) {
	3677	if (mpack_expect_array(reader) != count)
	3678	mpack_reader_flag_error(reader, mpack_error_type);
	3679	}
	3680
	3681	bool mpack_expect_array_or_nil(mpack_reader_t* reader, uint32_t* count) {
	3682	mpack_assert(count != NULL, "count cannot be NULL");
	3683
	3684	mpack_tag_t var = mpack_read_tag(reader);
	3685	if (var.type == mpack_type_nil) {
	3686	*count = 0;
	3687	return false;
	3688	}
	3689	if (var.type == mpack_type_array) {
	3690	*count = var.v.n;
	3691	return true;
	3692	}
	3693	mpack_reader_flag_error(reader, mpack_error_type);
	3694	*count = 0;
	3695	return false;
	3696	}
	3697
	3698	bool mpack_expect_array_max_or_nil(mpack_reader_t* reader, uint32_t max_count, uint32_t* count) {
	3699	mpack_assert(count != NULL, "count cannot be NULL");
	3700
	3701	bool has_array = mpack_expect_array_or_nil(reader, count);
	3702	if (has_array && *count > max_count) {
	3703	*count = 0;
	3704	mpack_reader_flag_error(reader, mpack_error_type);
	3705	return false;
	3706	}
	3707	return has_array;
	3708	}
	3709
	3710	#ifdef MPACK_MALLOC
	3711	void* mpack_expect_array_alloc_impl(mpack_reader_t* reader, size_t element_size, uint32_t max_count, uint32_t* out_count, bool allow_nil) {
	3712	mpack_assert(out_count != NULL, "out_count cannot be NULL");
	3713	*out_count = 0;
	3714
	3715	uint32_t count;
	3716	bool has_array = true;
	3717	if (allow_nil)
	3718	has_array = mpack_expect_array_max_or_nil(reader, max_count, &count);
	3719	else
	3720	count = mpack_expect_array_max(reader, max_count);
	3721	if (mpack_reader_error(reader))
	3722	return NULL;
	3723
	3724	// size 0 is not an error; we return NULL for no elements.
	3725	if (count == 0) {
	3726	// we call mpack_done_array() automatically ONLY if we are using
	3727	// the _or_nil variant. this is the only way to allow nil and empty
	3728	// to work the same way.
	3729	if (allow_nil && has_array)
	3730	mpack_done_array(reader);
	3731	return NULL;
	3732	}
	3733
	3734	void* p = MPACK_MALLOC(element_size * count);
	3735	if (p == NULL) {
	3736	mpack_reader_flag_error(reader, mpack_error_memory);
	3737	return NULL;
	3738	}
	3739
	3740	*out_count = count;
	3741	return p;
	3742	}
	3743	#endif
	3744
	3745
	3746	// Str, Bin and Ext Functions
	3747
	3748	uint32_t mpack_expect_str(mpack_reader_t* reader) {
	3749	#if MPACK_OPTIMIZE_FOR_SIZE
	3750	mpack_tag_t var = mpack_read_tag(reader);
	3751	if (var.type == mpack_type_str)
	3752	return var.v.l;
	3753	mpack_reader_flag_error(reader, mpack_error_type);
	3754	return 0;
	3755	#else
	3756	uint8_t type = mpack_expect_type_byte(reader);
	3757	uint32_t count;
	3758
	3759	if ((type >> 5) == 5) {
	3760	count = type & (uint8_t)~0xe0;
	3761	} else if (type == 0xd9) {
	3762	count = mpack_expect_native_u8(reader);
	3763	} else if (type == 0xda) {
	3764	count = mpack_expect_native_u16(reader);
	3765	} else if (type == 0xdb) {
	3766	count = mpack_expect_native_u32(reader);
	3767	} else {
	3768	mpack_reader_flag_error(reader, mpack_error_type);
	3769	return 0;
	3770	}
	3771
	3772	#if MPACK_READ_TRACKING
	3773	mpack_reader_flag_if_error(reader, mpack_track_push(&reader->track, mpack_type_str, count));
	3774	#endif
	3775	return count;
	3776	#endif
	3777	}
	3778
	3779	size_t mpack_expect_str_buf(mpack_reader_t* reader, char* buf, size_t bufsize) {
	3780	mpack_assert(buf != NULL, "buf cannot be NULL");
	3781
	3782	size_t length = mpack_expect_str(reader);
	3783	if (mpack_reader_error(reader))
	3784	return 0;
	3785
	3786	if (length > bufsize) {
	3787	mpack_reader_flag_error(reader, mpack_error_too_big);
	3788	return 0;
	3789	}
	3790
	3791	mpack_read_bytes(reader, buf, length);
	3792	if (mpack_reader_error(reader))
	3793	return 0;
	3794
	3795	mpack_done_str(reader);
	3796	return length;
	3797	}
	3798
	3799	size_t mpack_expect_utf8(mpack_reader_t* reader, char* buf, size_t size) {
	3800	mpack_assert(buf != NULL, "buf cannot be NULL");
	3801
	3802	size_t length = mpack_expect_str_buf(reader, buf, size);
	3803
	3804	if (!mpack_utf8_check(buf, length)) {
	3805	mpack_reader_flag_error(reader, mpack_error_type);
	3806	return 0;
	3807	}
	3808
	3809	return length;
	3810	}
	3811
	3812	uint32_t mpack_expect_bin(mpack_reader_t* reader) {
	3813	mpack_tag_t var = mpack_read_tag(reader);
	3814	if (var.type == mpack_type_bin)
	3815	return var.v.l;
	3816	mpack_reader_flag_error(reader, mpack_error_type);
	3817	return 0;
	3818	}
	3819
	3820	size_t mpack_expect_bin_buf(mpack_reader_t* reader, char* buf, size_t bufsize) {
	3821	mpack_assert(buf != NULL, "buf cannot be NULL");
	3822
	3823	size_t binsize = mpack_expect_bin(reader);
	3824	if (mpack_reader_error(reader))
	3825	return 0;
	3826	if (binsize > bufsize) {
	3827	mpack_reader_flag_error(reader, mpack_error_too_big);
	3828	return 0;
	3829	}
	3830	mpack_read_bytes(reader, buf, binsize);
	3831	if (mpack_reader_error(reader))
	3832	return 0;
	3833	mpack_done_bin(reader);
	3834	return binsize;
	3835	}
	3836
	3837	#if MPACK_EXTENSIONS
	3838	uint32_t mpack_expect_ext(mpack_reader_t* reader, int8_t* type) {
	3839	mpack_tag_t var = mpack_read_tag(reader);
	3840	if (var.type == mpack_type_ext) {
	3841	*type = mpack_tag_ext_exttype(&var);
	3842	return mpack_tag_ext_length(&var);
	3843	}
	3844	*type = 0;
	3845	mpack_reader_flag_error(reader, mpack_error_type);
	3846	return 0;
	3847	}
	3848
	3849	size_t mpack_expect_ext_buf(mpack_reader_t* reader, int8_t* type, char* buf, size_t bufsize) {
	3850	mpack_assert(buf != NULL, "buf cannot be NULL");
	3851
	3852	size_t extsize = mpack_expect_ext(reader, type);
	3853	if (mpack_reader_error(reader))
	3854	return 0;
	3855	if (extsize > bufsize) {
	3856	*type = 0;
	3857	mpack_reader_flag_error(reader, mpack_error_too_big);
	3858	return 0;
	3859	}
	3860	mpack_read_bytes(reader, buf, extsize);
	3861	if (mpack_reader_error(reader)) {
	3862	*type = 0;
	3863	return 0;
	3864	}
	3865	mpack_done_ext(reader);
	3866	return extsize;
	3867	}
	3868	#endif
	3869
	3870	void mpack_expect_cstr(mpack_reader_t* reader, char* buf, size_t bufsize) {
	3871	uint32_t length = mpack_expect_str(reader);
	3872	mpack_read_cstr(reader, buf, bufsize, length);
	3873	mpack_done_str(reader);
	3874	}
	3875
	3876	void mpack_expect_utf8_cstr(mpack_reader_t* reader, char* buf, size_t bufsize) {
	3877	uint32_t length = mpack_expect_str(reader);
	3878	mpack_read_utf8_cstr(reader, buf, bufsize, length);
	3879	mpack_done_str(reader);
	3880	}
	3881
	3882	#ifdef MPACK_MALLOC
	3883	static char* mpack_expect_cstr_alloc_unchecked(mpack_reader_t* reader, size_t maxsize, size_t* out_length) {
	3884	mpack_assert(out_length != NULL, "out_length cannot be NULL");
	3885	*out_length = 0;
	3886
	3887	// make sure argument makes sense
	3888	if (maxsize < 1) {
	3889	mpack_break("maxsize is zero; you must have room for at least a null-terminator");
	3890	mpack_reader_flag_error(reader, mpack_error_bug);
	3891	return NULL;
	3892	}
	3893
	3894	if (maxsize > UINT32_MAX)
	3895	maxsize = UINT32_MAX;
	3896
	3897	size_t length = mpack_expect_str_max(reader, (uint32_t)maxsize - 1);
	3898	char* str = mpack_read_bytes_alloc_impl(reader, length, true);
	3899	mpack_done_str(reader);
	3900
	3901	if (str)
	3902	*out_length = length;
	3903	return str;
	3904	}
	3905
	3906	char* mpack_expect_cstr_alloc(mpack_reader_t* reader, size_t maxsize) {
	3907	size_t length;
	3908	char* str = mpack_expect_cstr_alloc_unchecked(reader, maxsize, &length);
	3909
	3910	if (str && !mpack_str_check_no_null(str, length)) {
	3911	MPACK_FREE(str);
	3912	mpack_reader_flag_error(reader, mpack_error_type);
	3913	return NULL;
	3914	}
	3915
	3916	return str;
	3917	}
	3918
	3919	char* mpack_expect_utf8_cstr_alloc(mpack_reader_t* reader, size_t maxsize) {
	3920	size_t length;
	3921	char* str = mpack_expect_cstr_alloc_unchecked(reader, maxsize, &length);
	3922
	3923	if (str && !mpack_utf8_check_no_null(str, length)) {
	3924	MPACK_FREE(str);
	3925	mpack_reader_flag_error(reader, mpack_error_type);
	3926	return NULL;
	3927	}
	3928
	3929	return str;
	3930	}
	3931	#endif
	3932
	3933	void mpack_expect_str_match(mpack_reader_t* reader, const char* str, size_t len) {
	3934	mpack_assert(str != NULL, "str cannot be NULL");
	3935
	3936	// expect a str the correct length
	3937	if (len > UINT32_MAX)
	3938	mpack_reader_flag_error(reader, mpack_error_type);
	3939	mpack_expect_str_length(reader, (uint32_t)len);
	3940	if (mpack_reader_error(reader))
	3941	return;
	3942	mpack_reader_track_bytes(reader, len);
	3943
	3944	// check each byte one by one (matched strings are likely to be very small)
	3945	for (; len > 0; --len) {
	3946	if (mpack_expect_native_u8(reader) != *str++) {
	3947	mpack_reader_flag_error(reader, mpack_error_type);
	3948	return;
	3949	}
	3950	}
	3951
	3952	mpack_done_str(reader);
	3953	}
	3954
	3955	void mpack_expect_tag(mpack_reader_t* reader, mpack_tag_t expected) {
	3956	mpack_tag_t actual = mpack_read_tag(reader);
	3957	if (!mpack_tag_equal(actual, expected))
	3958	mpack_reader_flag_error(reader, mpack_error_type);
	3959	}
	3960
	3961	#ifdef MPACK_MALLOC
	3962	char* mpack_expect_bin_alloc(mpack_reader_t* reader, size_t maxsize, size_t* size) {
	3963	mpack_assert(size != NULL, "size cannot be NULL");
	3964	*size = 0;
	3965
	3966	if (maxsize > UINT32_MAX)
	3967	maxsize = UINT32_MAX;
	3968
	3969	size_t length = mpack_expect_bin_max(reader, (uint32_t)maxsize);
	3970	if (mpack_reader_error(reader))
	3971	return NULL;
	3972
	3973	char* data = mpack_read_bytes_alloc(reader, length);
	3974	mpack_done_bin(reader);
	3975
	3976	if (data)
	3977	*size = length;
	3978	return data;
	3979	}
	3980	#endif
	3981
	3982	#if MPACK_EXTENSIONS && defined(MPACK_MALLOC)
	3983	char* mpack_expect_ext_alloc(mpack_reader_t* reader, int8_t* type, size_t maxsize, size_t* size) {
	3984	mpack_assert(size != NULL, "size cannot be NULL");
	3985	*size = 0;
	3986
	3987	if (maxsize > UINT32_MAX)
	3988	maxsize = UINT32_MAX;
	3989
	3990	size_t length = mpack_expect_ext_max(reader, type, (uint32_t)maxsize);
	3991	if (mpack_reader_error(reader))
	3992	return NULL;
	3993
	3994	char* data = mpack_read_bytes_alloc(reader, length);
	3995	mpack_done_ext(reader);
	3996
	3997	if (data) {
	3998	*size = length;
	3999	} else {
	4000	*type = 0;
	4001	}
	4002	return data;
	4003	}
	4004	#endif
	4005
	4006	size_t mpack_expect_enum(mpack_reader_t* reader, const char* strings[], size_t count) {
	4007
	4008	// read the string in-place
	4009	size_t keylen = mpack_expect_str(reader);
	4010	const char* key = mpack_read_bytes_inplace(reader, keylen);
	4011	mpack_done_str(reader);
	4012	if (mpack_reader_error(reader) != mpack_ok)
	4013	return count;
	4014
	4015	// find what key it matches
	4016	for (size_t i = 0; i < count; ++i) {
	4017	const char* other = strings[i];
	4018	size_t otherlen = mpack_strlen(other);
	4019	if (keylen == otherlen && mpack_memcmp(key, other, keylen) == 0)
	4020	return i;
	4021	}
	4022
	4023	// no matches
	4024	mpack_reader_flag_error(reader, mpack_error_type);
	4025	return count;
	4026	}
	4027
	4028	size_t mpack_expect_enum_optional(mpack_reader_t* reader, const char* strings[], size_t count) {
	4029	if (mpack_reader_error(reader) != mpack_ok)
	4030	return count;
	4031
	4032	mpack_assert(count != 0, "count cannot be zero; no strings are valid!");
	4033	mpack_assert(strings != NULL, "strings cannot be NULL");
	4034
	4035	// the key is only recognized if it is a string
	4036	if (mpack_peek_tag(reader).type != mpack_type_str) {
	4037	mpack_discard(reader);
	4038	return count;
	4039	}
	4040
	4041	// read the string in-place
	4042	size_t keylen = mpack_expect_str(reader);
	4043	const char* key = mpack_read_bytes_inplace(reader, keylen);
	4044	mpack_done_str(reader);
	4045	if (mpack_reader_error(reader) != mpack_ok)
	4046	return count;
	4047
	4048	// find what key it matches
	4049	for (size_t i = 0; i < count; ++i) {
	4050	const char* other = strings[i];
	4051	size_t otherlen = mpack_strlen(other);
	4052	if (keylen == otherlen && mpack_memcmp(key, other, keylen) == 0)
	4053	return i;
	4054	}
	4055
	4056	// no matches
	4057	return count;
	4058	}
	4059
	4060	size_t mpack_expect_key_uint(mpack_reader_t* reader, bool found[], size_t count) {
	4061	if (mpack_reader_error(reader) != mpack_ok)
	4062	return count;
	4063
	4064	if (count == 0) {
	4065	mpack_break("count cannot be zero; no keys are valid!");
	4066	mpack_reader_flag_error(reader, mpack_error_bug);
	4067	return count;
	4068	}
	4069	mpack_assert(found != NULL, "found cannot be NULL");
	4070
	4071	// the key is only recognized if it is an unsigned int
	4072	if (mpack_peek_tag(reader).type != mpack_type_uint) {
	4073	mpack_discard(reader);
	4074	return count;
	4075	}
	4076
	4077	// read the key
	4078	uint64_t value = mpack_expect_u64(reader);
	4079	if (mpack_reader_error(reader) != mpack_ok)
	4080	return count;
	4081
	4082	// unrecognized keys are fine, we just return count
	4083	if (value >= count)
	4084	return count;
	4085
	4086	// check if this key is a duplicate
	4087	if (found[value]) {
	4088	mpack_reader_flag_error(reader, mpack_error_invalid);
	4089	return count;
	4090	}
	4091
	4092	found[value] = true;
	4093	return (size_t)value;
	4094	}
	4095
	4096	size_t mpack_expect_key_cstr(mpack_reader_t* reader, const char* keys[], bool found[], size_t count) {
	4097	size_t i = mpack_expect_enum_optional(reader, keys, count);
	4098
	4099	// unrecognized keys are fine, we just return count
	4100	if (i == count)
	4101	return count;
	4102
	4103	// check if this key is a duplicate
	4104	mpack_assert(found != NULL, "found cannot be NULL");
	4105	if (found[i]) {
	4106	mpack_reader_flag_error(reader, mpack_error_invalid);
	4107	return count;
	4108	}
	4109
	4110	found[i] = true;
	4111	return i;
	4112	}
	4113
	4114	#endif
	4115
	4116
	4117	/* mpack/mpack-node.c.c */
	4118
	4119	#define MPACK_INTERNAL 1
	4120
	4121	/* #include "mpack-node.h" */
	4122
	4123	#if MPACK_NODE
	4124
	4125	MPACK_STATIC_INLINE const char* mpack_node_data_unchecked(mpack_node_t node) {
	4126	mpack_assert(mpack_node_error(node) == mpack_ok, "tree is in an error state!");
	4127
	4128	mpack_type_t type = node.data->type;
	4129	MPACK_UNUSED(type);
	4130	#if MPACK_EXTENSIONS
	4131	mpack_assert(type == mpack_type_str \|\| type == mpack_type_bin \|\| type == mpack_type_ext,
	4132	"node of type %i (%s) is not a data type!", type, mpack_type_to_string(type));
	4133	#else
	4134	mpack_assert(type == mpack_type_str \|\| type == mpack_type_bin,
	4135	"node of type %i (%s) is not a data type!", type, mpack_type_to_string(type));
	4136	#endif
	4137
	4138	return node.tree->data + node.data->value.offset;
	4139	}
	4140
	4141	#if MPACK_EXTENSIONS
	4142	MPACK_STATIC_INLINE int8_t mpack_node_exttype_unchecked(mpack_node_t node) {
	4143	mpack_assert(mpack_node_error(node) == mpack_ok, "tree is in an error state!");
	4144
	4145	mpack_type_t type = node.data->type;
	4146	MPACK_UNUSED(type);
	4147	mpack_assert(type == mpack_type_ext, "node of type %i (%s) is not an ext type!",
	4148	type, mpack_type_to_string(type));
	4149
	4150	// the exttype of an ext node is stored in the byte preceding the data
	4151	return mpack_load_i8(mpack_node_data_unchecked(node) - 1);
	4152	}
	4153	#endif
	4154
	4155
	4156
	4157	/*
	4158	* Tree Parsing
	4159	*/
	4160
	4161	#ifdef MPACK_MALLOC
	4162
	4163	// fix up the alloc size to make sure it exactly fits the
	4164	// maximum number of nodes it can contain (the allocator will
	4165	// waste it back anyway, but we round it down just in case)
	4166
	4167	#define MPACK_NODES_PER_PAGE \
	4168	((MPACK_NODE_PAGE_SIZE - sizeof(mpack_tree_page_t)) / sizeof(mpack_node_data_t) + 1)
	4169
	4170	#define MPACK_PAGE_ALLOC_SIZE \
	4171	(sizeof(mpack_tree_page_t) + sizeof(mpack_node_data_t) * (MPACK_NODES_PER_PAGE - 1))
	4172
	4173	#endif
	4174
	4175	#ifdef MPACK_MALLOC
	4176	/*
	4177	* Fills the tree until we have at least enough bytes for the current node.
	4178	*/
	4179	static bool mpack_tree_reserve_fill(mpack_tree_t* tree) {
	4180	mpack_assert(tree->parser.state == mpack_tree_parse_state_in_progress);
	4181
	4182	size_t bytes = tree->parser.current_node_reserved;
	4183	mpack_assert(bytes > tree->parser.possible_nodes_left,
	4184	"there are already enough bytes! call mpack_tree_ensure() instead.");
	4185	mpack_log("filling to reserve %i bytes\n", (int)bytes);
	4186
	4187	// if the necessary bytes would put us over the maximum tree
	4188	// size, fail right away.
	4189	// TODO: check for overflow?
	4190	if (tree->data_length + bytes > tree->max_size) {
	4191	mpack_tree_flag_error(tree, mpack_error_too_big);
	4192	return false;
	4193	}
	4194
	4195	// we'll need a read function to fetch more data. if there's
	4196	// no read function, the data should contain an entire message
	4197	// (or messages), so we flag it as invalid.
	4198	if (tree->read_fn == NULL) {
	4199	mpack_log("tree has no read function!\n");
	4200	mpack_tree_flag_error(tree, mpack_error_invalid);
	4201	return false;
	4202	}
	4203
	4204	// expand the buffer if needed
	4205	if (tree->data_length + bytes > tree->buffer_capacity) {
	4206
	4207	// TODO: check for overflow?
	4208	size_t new_capacity = (tree->buffer_capacity == 0) ? MPACK_BUFFER_SIZE : tree->buffer_capacity;
	4209	while (new_capacity < tree->data_length + bytes)
	4210	new_capacity *= 2;
	4211	if (new_capacity > tree->max_size)
	4212	new_capacity = tree->max_size;
	4213
	4214	mpack_log("expanding buffer from %i to %i\n", (int)tree->buffer_capacity, (int)new_capacity);
	4215
	4216	char* new_buffer;
	4217	if (tree->buffer == NULL)
	4218	new_buffer = (char*)MPACK_MALLOC(new_capacity);
	4219	else
	4220	new_buffer = (char*)mpack_realloc(tree->buffer, tree->data_length, new_capacity);
	4221
	4222	if (new_buffer == NULL) {
	4223	mpack_tree_flag_error(tree, mpack_error_memory);
	4224	return false;
	4225	}
	4226
	4227	tree->data = new_buffer;
	4228	tree->buffer = new_buffer;
	4229	tree->buffer_capacity = new_capacity;
	4230	}
	4231
	4232	// request as much data as possible, looping until we have
	4233	// all the data we need
	4234	do {
	4235	size_t read = tree->read_fn(tree, tree->buffer + tree->data_length, tree->buffer_capacity - tree->data_length);
	4236
	4237	// If the fill function encounters an error, it should flag an error on
	4238	// the tree.
	4239	if (mpack_tree_error(tree) != mpack_ok)
	4240	return false;
	4241
	4242	// We guard against fill functions that return -1 just in case.
	4243	if (read == (size_t)(-1)) {
	4244	mpack_tree_flag_error(tree, mpack_error_io);
	4245	return false;
	4246	}
	4247
	4248	// If the fill function returns 0, the data is not available yet. We
	4249	// return false to stop parsing the current node.
	4250	if (read == 0) {
	4251	mpack_log("not enough data.\n");
	4252	return false;
	4253	}
	4254
	4255	mpack_log("read %u more bytes\n", (uint32_t)read);
	4256	tree->data_length += read;
	4257	tree->parser.possible_nodes_left += read;
	4258	} while (tree->parser.possible_nodes_left < bytes);
	4259
	4260	return true;
	4261	}
	4262	#endif
	4263
	4264	/*
	4265	* Ensures there are enough additional bytes in the tree for the current node
	4266	* (including reserved bytes for the children of this node, and in addition to
	4267	* the reserved bytes for children of previous compound nodes), reading more
	4268	* data if needed.
	4269	*
	4270	* extra_bytes is the number of additional bytes to reserve for the current
	4271	* node beyond the type byte (since one byte is already reserved for each node
	4272	* by its parent array or map.)
	4273	*
	4274	* This may reallocate the tree, which means the tree->data pointer may change!
	4275	*
	4276	* Returns false if not enough bytes could be read.
	4277	*/
	4278	MPACK_STATIC_INLINE bool mpack_tree_reserve_bytes(mpack_tree_t* tree, size_t extra_bytes) {
	4279	mpack_assert(tree->parser.state == mpack_tree_parse_state_in_progress);
	4280
	4281	// We guard against overflow here. A compound type could declare more than
	4282	// UINT32_MAX contents which overflows SIZE_MAX on 32-bit platforms. We
	4283	// flag mpack_error_invalid instead of mpack_error_too_big since it's far
	4284	// more likely that the message is corrupt than that the data is valid but
	4285	// not parseable on this architecture (see test_read_node_possible() in
	4286	// test-node.c .)
	4287	if ((uint64_t)tree->parser.current_node_reserved + (uint64_t)extra_bytes > SIZE_MAX) {
	4288	mpack_tree_flag_error(tree, mpack_error_invalid);
	4289	return false;
	4290	}
	4291
	4292	tree->parser.current_node_reserved += extra_bytes;
	4293
	4294	// Note that possible_nodes_left already accounts for reserved bytes for
	4295	// children of previous compound nodes. So even if there are hundreds of
	4296	// bytes left in the buffer, we might need to read anyway.
	4297	if (tree->parser.current_node_reserved <= tree->parser.possible_nodes_left)
	4298	return true;
	4299
	4300	#ifdef MPACK_MALLOC
	4301	return mpack_tree_reserve_fill(tree);
	4302	#else
	4303	return false;
	4304	#endif
	4305	}
	4306
	4307	MPACK_STATIC_INLINE size_t mpack_tree_parser_stack_capacity(mpack_tree_t* tree) {
	4308	#ifdef MPACK_MALLOC
	4309	return tree->parser.stack_capacity;
	4310	#else
	4311	return sizeof(tree->parser.stack) / sizeof(tree->parser.stack[0]);
	4312	#endif
	4313	}
	4314
	4315	static bool mpack_tree_push_stack(mpack_tree_t* tree, mpack_node_data_t* first_child, size_t total) {
	4316	mpack_tree_parser_t* parser = &tree->parser;
	4317	mpack_assert(parser->state == mpack_tree_parse_state_in_progress);
	4318
	4319	// No need to push empty containers
	4320	if (total == 0)
	4321	return true;
	4322
	4323	// Make sure we have enough room in the stack
	4324	if (parser->level + 1 == mpack_tree_parser_stack_capacity(tree)) {
	4325	#ifdef MPACK_MALLOC
	4326	size_t new_capacity = parser->stack_capacity * 2;
	4327	mpack_log("growing parse stack to capacity %i\n", (int)new_capacity);
	4328
	4329	// Replace the stack-allocated parsing stack
	4330	if (!parser->stack_owned) {
	4331	mpack_level_t* new_stack = (mpack_level_t)MPACK_MALLOC(sizeof(mpack_level_t) new_capacity);
	4332	if (!new_stack) {
	4333	mpack_tree_flag_error(tree, mpack_error_memory);
	4334	return false;
	4335	}
	4336	mpack_memcpy(new_stack, parser->stack, sizeof(mpack_level_t) * parser->stack_capacity);
	4337	parser->stack = new_stack;
	4338	parser->stack_owned = true;
	4339
	4340	// Realloc the allocated parsing stack
	4341	} else {
	4342	mpack_level_t* new_stack = (mpack_level_t*)mpack_realloc(parser->stack,
	4343	sizeof(mpack_level_t) * parser->stack_capacity, sizeof(mpack_level_t) * new_capacity);
	4344	if (!new_stack) {
	4345	mpack_tree_flag_error(tree, mpack_error_memory);
	4346	return false;
	4347	}
	4348	parser->stack = new_stack;
	4349	}
	4350	parser->stack_capacity = new_capacity;
	4351	#else
	4352	mpack_tree_flag_error(tree, mpack_error_too_big);
	4353	return false;
	4354	#endif
	4355	}
	4356
	4357	// Push the contents of this node onto the parsing stack
	4358	++parser->level;
	4359	parser->stack[parser->level].child = first_child;
	4360	parser->stack[parser->level].left = total;
	4361	return true;
	4362	}
	4363
	4364	static bool mpack_tree_parse_children(mpack_tree_t* tree, mpack_node_data_t* node) {
	4365	mpack_tree_parser_t* parser = &tree->parser;
	4366	mpack_assert(parser->state == mpack_tree_parse_state_in_progress);
	4367
	4368	mpack_type_t type = node->type;
	4369	size_t total = node->len;
	4370
	4371	// Calculate total elements to read
	4372	if (type == mpack_type_map) {
	4373	if ((uint64_t)total * 2 > SIZE_MAX) {
	4374	mpack_tree_flag_error(tree, mpack_error_too_big);
	4375	return false;
	4376	}
	4377	total *= 2;
	4378	}
	4379
	4380	// Make sure we are under our total node limit (TODO can this overflow?)
	4381	tree->node_count += total;
	4382	if (tree->node_count > tree->max_nodes) {
	4383	mpack_tree_flag_error(tree, mpack_error_too_big);
	4384	return false;
	4385	}
	4386
	4387	// Each node is at least one byte. Count these bytes now to make
	4388	// sure there is enough data left.
	4389	if (!mpack_tree_reserve_bytes(tree, total))
	4390	return false;
	4391
	4392	// If there are enough nodes left in the current page, no need to grow
	4393	if (total <= parser->nodes_left) {
	4394	node->value.children = parser->nodes;
	4395	parser->nodes += total;
	4396	parser->nodes_left -= total;
	4397
	4398	} else {
	4399
	4400	#ifdef MPACK_MALLOC
	4401
	4402	// We can't grow if we're using a fixed pool (i.e. we didn't start with a page)
	4403	if (!tree->next) {
	4404	mpack_tree_flag_error(tree, mpack_error_too_big);
	4405	return false;
	4406	}
	4407
	4408	// Otherwise we need to grow, and the node's children need to be contiguous.
	4409	// This is a heuristic to decide whether we should waste the remaining space
	4410	// in the current page and start a new one, or give the children their
	4411	// own page. With a fraction of 1/8, this causes at most 12% additional
	4412	// waste. Note that reducing this too much causes less cache coherence and
	4413	// more malloc() overhead due to smaller allocations, so there's a tradeoff
	4414	// here. This heuristic could use some improvement, especially with custom
	4415	// page sizes.
	4416
	4417	mpack_tree_page_t* page;
	4418
	4419	if (total > MPACK_NODES_PER_PAGE \|\| parser->nodes_left > MPACK_NODES_PER_PAGE / 8) {
	4420	// TODO: this should check for overflow
	4421	page = (mpack_tree_page_t*)MPACK_MALLOC(
	4422	sizeof(mpack_tree_page_t) + sizeof(mpack_node_data_t) * (total - 1));
	4423	if (page == NULL) {
	4424	mpack_tree_flag_error(tree, mpack_error_memory);
	4425	return false;
	4426	}
	4427	mpack_log("allocated seperate page %p for %i children, %i left in page of %i total\n",
	4428	page, (int)total, (int)parser->nodes_left, (int)MPACK_NODES_PER_PAGE);
	4429
	4430	node->value.children = page->nodes;
	4431
	4432	} else {
	4433	page = (mpack_tree_page_t*)MPACK_MALLOC(MPACK_PAGE_ALLOC_SIZE);
	4434	if (page == NULL) {
	4435	mpack_tree_flag_error(tree, mpack_error_memory);
	4436	return false;
	4437	}
	4438	mpack_log("allocated new page %p for %i children, wasting %i in page of %i total\n",
	4439	page, (int)total, (int)parser->nodes_left, (int)MPACK_NODES_PER_PAGE);
	4440
	4441	node->value.children = page->nodes;
	4442	parser->nodes = page->nodes + total;
	4443	parser->nodes_left = MPACK_NODES_PER_PAGE - total;
	4444	}
	4445
	4446	page->next = tree->next;
	4447	tree->next = page;
	4448
	4449	#else
	4450	// We can't grow if we don't have an allocator
	4451	mpack_tree_flag_error(tree, mpack_error_too_big);
	4452	return false;
	4453	#endif
	4454	}
	4455
	4456	return mpack_tree_push_stack(tree, node->value.children, total);
	4457	}
	4458
	4459	static bool mpack_tree_parse_bytes(mpack_tree_t* tree, mpack_node_data_t* node) {
	4460	node->value.offset = tree->size + tree->parser.current_node_reserved + 1;
	4461	return mpack_tree_reserve_bytes(tree, node->len);
	4462	}
	4463
	4464	#if MPACK_EXTENSIONS
	4465	static bool mpack_tree_parse_ext(mpack_tree_t* tree, mpack_node_data_t* node) {
	4466	// reserve space for exttype
	4467	tree->parser.current_node_reserved += sizeof(int8_t);
	4468	node->type = mpack_type_ext;
	4469	return mpack_tree_parse_bytes(tree, node);
	4470	}
	4471	#endif
	4472
	4473	static bool mpack_tree_parse_node_contents(mpack_tree_t* tree, mpack_node_data_t* node) {
	4474	mpack_assert(tree->parser.state == mpack_tree_parse_state_in_progress);
	4475	mpack_assert(node != NULL, "null node?");
	4476
	4477	// read the type. we've already accounted for this byte in
	4478	// possible_nodes_left, so we already know it is in bounds, and we don't
	4479	// need to reserve it for this node.
	4480	mpack_assert(tree->data_length > tree->size);
	4481	uint8_t type = mpack_load_u8(tree->data + tree->size);
	4482	mpack_log("node type %x\n", type);
	4483	tree->parser.current_node_reserved = 0;
	4484
	4485	// as with mpack_read_tag(), the fastest way to parse a node is to switch
	4486	// on the first byte, and to explicitly list every possible byte. we switch
	4487	// on the first four bits in size-optimized builds.
	4488
	4489	#if MPACK_OPTIMIZE_FOR_SIZE
	4490	switch (type >> 4) {
	4491
	4492	// positive fixnum
	4493	case 0x0: case 0x1: case 0x2: case 0x3:
	4494	case 0x4: case 0x5: case 0x6: case 0x7:
	4495	node->type = mpack_type_uint;
	4496	node->value.u = type;
	4497	return true;
	4498
	4499	// negative fixnum
	4500	case 0xe: case 0xf:
	4501	node->type = mpack_type_int;
	4502	node->value.i = (int8_t)type;
	4503	return true;
	4504
	4505	// fixmap
	4506	case 0x8:
	4507	node->type = mpack_type_map;
	4508	node->len = (uint32_t)(type & ~0xf0);
	4509	return mpack_tree_parse_children(tree, node);
	4510
	4511	// fixarray
	4512	case 0x9:
	4513	node->type = mpack_type_array;
	4514	node->len = (uint32_t)(type & ~0xf0);
	4515	return mpack_tree_parse_children(tree, node);
	4516
	4517	// fixstr
	4518	case 0xa: case 0xb:
	4519	node->type = mpack_type_str;
	4520	node->len = (uint32_t)(type & ~0xe0);
	4521	return mpack_tree_parse_bytes(tree, node);
	4522
	4523	// not one of the common infix types
	4524	default:
	4525	break;
	4526	}
	4527	#endif
	4528
	4529	switch (type) {
	4530
	4531	#if !MPACK_OPTIMIZE_FOR_SIZE
	4532	// positive fixnum
	4533	case 0x00: case 0x01: case 0x02: case 0x03: case 0x04: case 0x05: case 0x06: case 0x07:
	4534	case 0x08: case 0x09: case 0x0a: case 0x0b: case 0x0c: case 0x0d: case 0x0e: case 0x0f:
	4535	case 0x10: case 0x11: case 0x12: case 0x13: case 0x14: case 0x15: case 0x16: case 0x17:
	4536	case 0x18: case 0x19: case 0x1a: case 0x1b: case 0x1c: case 0x1d: case 0x1e: case 0x1f:
	4537	case 0x20: case 0x21: case 0x22: case 0x23: case 0x24: case 0x25: case 0x26: case 0x27:
	4538	case 0x28: case 0x29: case 0x2a: case 0x2b: case 0x2c: case 0x2d: case 0x2e: case 0x2f:
	4539	case 0x30: case 0x31: case 0x32: case 0x33: case 0x34: case 0x35: case 0x36: case 0x37:
	4540	case 0x38: case 0x39: case 0x3a: case 0x3b: case 0x3c: case 0x3d: case 0x3e: case 0x3f:
	4541	case 0x40: case 0x41: case 0x42: case 0x43: case 0x44: case 0x45: case 0x46: case 0x47:
	4542	case 0x48: case 0x49: case 0x4a: case 0x4b: case 0x4c: case 0x4d: case 0x4e: case 0x4f:
	4543	case 0x50: case 0x51: case 0x52: case 0x53: case 0x54: case 0x55: case 0x56: case 0x57:
	4544	case 0x58: case 0x59: case 0x5a: case 0x5b: case 0x5c: case 0x5d: case 0x5e: case 0x5f:
	4545	case 0x60: case 0x61: case 0x62: case 0x63: case 0x64: case 0x65: case 0x66: case 0x67:
	4546	case 0x68: case 0x69: case 0x6a: case 0x6b: case 0x6c: case 0x6d: case 0x6e: case 0x6f:
	4547	case 0x70: case 0x71: case 0x72: case 0x73: case 0x74: case 0x75: case 0x76: case 0x77:
	4548	case 0x78: case 0x79: case 0x7a: case 0x7b: case 0x7c: case 0x7d: case 0x7e: case 0x7f:
	4549	node->type = mpack_type_uint;
	4550	node->value.u = type;
	4551	return true;
	4552
	4553	// negative fixnum
	4554	case 0xe0: case 0xe1: case 0xe2: case 0xe3: case 0xe4: case 0xe5: case 0xe6: case 0xe7:
	4555	case 0xe8: case 0xe9: case 0xea: case 0xeb: case 0xec: case 0xed: case 0xee: case 0xef:
	4556	case 0xf0: case 0xf1: case 0xf2: case 0xf3: case 0xf4: case 0xf5: case 0xf6: case 0xf7:
	4557	case 0xf8: case 0xf9: case 0xfa: case 0xfb: case 0xfc: case 0xfd: case 0xfe: case 0xff:
	4558	node->type = mpack_type_int;
	4559	node->value.i = (int8_t)type;
	4560	return true;
	4561
	4562	// fixmap
	4563	case 0x80: case 0x81: case 0x82: case 0x83: case 0x84: case 0x85: case 0x86: case 0x87:
	4564	case 0x88: case 0x89: case 0x8a: case 0x8b: case 0x8c: case 0x8d: case 0x8e: case 0x8f:
	4565	node->type = mpack_type_map;
	4566	node->len = (uint32_t)(type & ~0xf0);
	4567	return mpack_tree_parse_children(tree, node);
	4568
	4569	// fixarray
	4570	case 0x90: case 0x91: case 0x92: case 0x93: case 0x94: case 0x95: case 0x96: case 0x97:
	4571	case 0x98: case 0x99: case 0x9a: case 0x9b: case 0x9c: case 0x9d: case 0x9e: case 0x9f:
	4572	node->type = mpack_type_array;
	4573	node->len = (uint32_t)(type & ~0xf0);
	4574	return mpack_tree_parse_children(tree, node);
	4575
	4576	// fixstr
	4577	case 0xa0: case 0xa1: case 0xa2: case 0xa3: case 0xa4: case 0xa5: case 0xa6: case 0xa7:
	4578	case 0xa8: case 0xa9: case 0xaa: case 0xab: case 0xac: case 0xad: case 0xae: case 0xaf:
	4579	case 0xb0: case 0xb1: case 0xb2: case 0xb3: case 0xb4: case 0xb5: case 0xb6: case 0xb7:
	4580	case 0xb8: case 0xb9: case 0xba: case 0xbb: case 0xbc: case 0xbd: case 0xbe: case 0xbf:
	4581	node->type = mpack_type_str;
	4582	node->len = (uint32_t)(type & ~0xe0);
	4583	return mpack_tree_parse_bytes(tree, node);
	4584	#endif
	4585
	4586	// nil
	4587	case 0xc0:
	4588	node->type = mpack_type_nil;
	4589	return true;
	4590
	4591	// bool
	4592	case 0xc2: case 0xc3:
	4593	node->type = mpack_type_bool;
	4594	node->value.b = type & 1;
	4595	return true;
	4596
	4597	// bin8
	4598	case 0xc4:
	4599	node->type = mpack_type_bin;
	4600	if (!mpack_tree_reserve_bytes(tree, sizeof(uint8_t)))
	4601	return false;
	4602	node->len = mpack_load_u8(tree->data + tree->size + 1);
	4603	return mpack_tree_parse_bytes(tree, node);
	4604
	4605	// bin16
	4606	case 0xc5:
	4607	node->type = mpack_type_bin;
	4608	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4609	return false;
	4610	node->len = mpack_load_u16(tree->data + tree->size + 1);
	4611	return mpack_tree_parse_bytes(tree, node);
	4612
	4613	// bin32
	4614	case 0xc6:
	4615	node->type = mpack_type_bin;
	4616	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4617	return false;
	4618	node->len = mpack_load_u32(tree->data + tree->size + 1);
	4619	return mpack_tree_parse_bytes(tree, node);
	4620
	4621	#if MPACK_EXTENSIONS
	4622	// ext8
	4623	case 0xc7:
	4624	if (!mpack_tree_reserve_bytes(tree, sizeof(uint8_t)))
	4625	return false;
	4626	node->len = mpack_load_u8(tree->data + tree->size + 1);
	4627	return mpack_tree_parse_ext(tree, node);
	4628
	4629	// ext16
	4630	case 0xc8:
	4631	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4632	return false;
	4633	node->len = mpack_load_u16(tree->data + tree->size + 1);
	4634	return mpack_tree_parse_ext(tree, node);
	4635
	4636	// ext32
	4637	case 0xc9:
	4638	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4639	return false;
	4640	node->len = mpack_load_u32(tree->data + tree->size + 1);
	4641	return mpack_tree_parse_ext(tree, node);
	4642	#endif
	4643
	4644	// float
	4645	case 0xca:
	4646	if (!mpack_tree_reserve_bytes(tree, sizeof(float)))
	4647	return false;
	4648	node->value.f = mpack_load_float(tree->data + tree->size + 1);
	4649	node->type = mpack_type_float;
	4650	return true;
	4651
	4652	// double
	4653	case 0xcb:
	4654	if (!mpack_tree_reserve_bytes(tree, sizeof(double)))
	4655	return false;
	4656	node->value.d = mpack_load_double(tree->data + tree->size + 1);
	4657	node->type = mpack_type_double;
	4658	return true;
	4659
	4660	// uint8
	4661	case 0xcc:
	4662	node->type = mpack_type_uint;
	4663	if (!mpack_tree_reserve_bytes(tree, sizeof(uint8_t)))
	4664	return false;
	4665	node->value.u = mpack_load_u8(tree->data + tree->size + 1);
	4666	return true;
	4667
	4668	// uint16
	4669	case 0xcd:
	4670	node->type = mpack_type_uint;
	4671	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4672	return false;
	4673	node->value.u = mpack_load_u16(tree->data + tree->size + 1);
	4674	return true;
	4675
	4676	// uint32
	4677	case 0xce:
	4678	node->type = mpack_type_uint;
	4679	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4680	return false;
	4681	node->value.u = mpack_load_u32(tree->data + tree->size + 1);
	4682	return true;
	4683
	4684	// uint64
	4685	case 0xcf:
	4686	node->type = mpack_type_uint;
	4687	if (!mpack_tree_reserve_bytes(tree, sizeof(uint64_t)))
	4688	return false;
	4689	node->value.u = mpack_load_u64(tree->data + tree->size + 1);
	4690	return true;
	4691
	4692	// int8
	4693	case 0xd0:
	4694	node->type = mpack_type_int;
	4695	if (!mpack_tree_reserve_bytes(tree, sizeof(int8_t)))
	4696	return false;
	4697	node->value.i = mpack_load_i8(tree->data + tree->size + 1);
	4698	return true;
	4699
	4700	// int16
	4701	case 0xd1:
	4702	node->type = mpack_type_int;
	4703	if (!mpack_tree_reserve_bytes(tree, sizeof(int16_t)))
	4704	return false;
	4705	node->value.i = mpack_load_i16(tree->data + tree->size + 1);
	4706	return true;
	4707
	4708	// int32
	4709	case 0xd2:
	4710	node->type = mpack_type_int;
	4711	if (!mpack_tree_reserve_bytes(tree, sizeof(int32_t)))
	4712	return false;
	4713	node->value.i = mpack_load_i32(tree->data + tree->size + 1);
	4714	return true;
	4715
	4716	// int64
	4717	case 0xd3:
	4718	node->type = mpack_type_int;
	4719	if (!mpack_tree_reserve_bytes(tree, sizeof(int64_t)))
	4720	return false;
	4721	node->value.i = mpack_load_i64(tree->data + tree->size + 1);
	4722	return true;
	4723
	4724	#if MPACK_EXTENSIONS
	4725	// fixext1
	4726	case 0xd4:
	4727	node->len = 1;
	4728	return mpack_tree_parse_ext(tree, node);
	4729
	4730	// fixext2
	4731	case 0xd5:
	4732	node->len = 2;
	4733	return mpack_tree_parse_ext(tree, node);
	4734
	4735	// fixext4
	4736	case 0xd6:
	4737	node->len = 4;
	4738	return mpack_tree_parse_ext(tree, node);
	4739
	4740	// fixext8
	4741	case 0xd7:
	4742	node->len = 8;
	4743	return mpack_tree_parse_ext(tree, node);
	4744
	4745	// fixext16
	4746	case 0xd8:
	4747	node->len = 16;
	4748	return mpack_tree_parse_ext(tree, node);
	4749	#endif
	4750
	4751	// str8
	4752	case 0xd9:
	4753	if (!mpack_tree_reserve_bytes(tree, sizeof(uint8_t)))
	4754	return false;
	4755	node->len = mpack_load_u8(tree->data + tree->size + 1);
	4756	node->type = mpack_type_str;
	4757	return mpack_tree_parse_bytes(tree, node);
	4758
	4759	// str16
	4760	case 0xda:
	4761	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4762	return false;
	4763	node->len = mpack_load_u16(tree->data + tree->size + 1);
	4764	node->type = mpack_type_str;
	4765	return mpack_tree_parse_bytes(tree, node);
	4766
	4767	// str32
	4768	case 0xdb:
	4769	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4770	return false;
	4771	node->len = mpack_load_u32(tree->data + tree->size + 1);
	4772	node->type = mpack_type_str;
	4773	return mpack_tree_parse_bytes(tree, node);
	4774
	4775	// array16
	4776	case 0xdc:
	4777	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4778	return false;
	4779	node->len = mpack_load_u16(tree->data + tree->size + 1);
	4780	node->type = mpack_type_array;
	4781	return mpack_tree_parse_children(tree, node);
	4782
	4783	// array32
	4784	case 0xdd:
	4785	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4786	return false;
	4787	node->len = mpack_load_u32(tree->data + tree->size + 1);
	4788	node->type = mpack_type_array;
	4789	return mpack_tree_parse_children(tree, node);
	4790
	4791	// map16
	4792	case 0xde:
	4793	if (!mpack_tree_reserve_bytes(tree, sizeof(uint16_t)))
	4794	return false;
	4795	node->len = mpack_load_u16(tree->data + tree->size + 1);
	4796	node->type = mpack_type_map;
	4797	return mpack_tree_parse_children(tree, node);
	4798
	4799	// map32
	4800	case 0xdf:
	4801	if (!mpack_tree_reserve_bytes(tree, sizeof(uint32_t)))
	4802	return false;
	4803	node->len = mpack_load_u32(tree->data + tree->size + 1);
	4804	node->type = mpack_type_map;
	4805	return mpack_tree_parse_children(tree, node);
	4806
	4807	// reserved
	4808	case 0xc1:
	4809	mpack_tree_flag_error(tree, mpack_error_invalid);
	4810	return false;
	4811
	4812	#if !MPACK_EXTENSIONS
	4813	// ext
	4814	case 0xc7: // fallthrough
	4815	case 0xc8: // fallthrough
	4816	case 0xc9: // fallthrough
	4817	// fixext
	4818	case 0xd4: // fallthrough
	4819	case 0xd5: // fallthrough
	4820	case 0xd6: // fallthrough
	4821	case 0xd7: // fallthrough
	4822	case 0xd8:
	4823	mpack_tree_flag_error(tree, mpack_error_unsupported);
	4824	return false;
	4825	#endif
	4826
	4827	#if MPACK_OPTIMIZE_FOR_SIZE
	4828	// any other bytes should have been handled by the infix switch
	4829	default:
	4830	break;
	4831	#endif
	4832	}
	4833
	4834	mpack_assert(0, "unreachable");
	4835	return false;
	4836	}
	4837
	4838	static bool mpack_tree_parse_node(mpack_tree_t* tree, mpack_node_data_t* node) {
	4839	mpack_log("parsing a node at position %i in level %i\n",
	4840	(int)tree->size, (int)tree->parser.level);
	4841
	4842	if (!mpack_tree_parse_node_contents(tree, node)) {
	4843	mpack_log("node parsing returned false\n");
	4844	return false;
	4845	}
	4846
	4847	tree->parser.possible_nodes_left -= tree->parser.current_node_reserved;
	4848
	4849	// The reserve for the current node does not include the initial byte
	4850	// previously reserved as part of its parent.
	4851	size_t node_size = tree->parser.current_node_reserved + 1;
	4852
	4853	// If the parsed type is a map or array, the reserve includes one byte for
	4854	// each child. We want to subtract these out of possible_nodes_left, but
	4855	// not out of the current size of the tree.
	4856	if (node->type == mpack_type_array)
	4857	node_size -= node->len;
	4858	else if (node->type == mpack_type_map)
	4859	node_size -= node->len * 2;
	4860	tree->size += node_size;
	4861
	4862	mpack_log("parsed a node of type %s of %i bytes and "
	4863	"%i additional bytes reserved for children.\n",
	4864	mpack_type_to_string(node->type), (int)node_size,
	4865	(int)tree->parser.current_node_reserved + 1 - (int)node_size);
	4866
	4867	return true;
	4868	}
	4869
	4870	/*
	4871	* We read nodes in a loop instead of recursively for maximum performance. The
	4872	* stack holds the amount of children left to read in each level of the tree.
	4873	* Parsing can pause and resume when more data becomes available.
	4874	*/
	4875	static bool mpack_tree_continue_parsing(mpack_tree_t* tree) {
	4876	if (mpack_tree_error(tree) != mpack_ok)
	4877	return false;
	4878
	4879	mpack_tree_parser_t* parser = &tree->parser;
	4880	mpack_assert(parser->state == mpack_tree_parse_state_in_progress);
	4881	mpack_log("parsing tree elements, %i bytes in buffer\n", (int)tree->data_length);
	4882
	4883	// we loop parsing nodes until the parse stack is empty. we break
	4884	// by returning out of the function.
	4885	while (true) {
	4886	mpack_node_data_t* node = parser->stack[parser->level].child;
	4887	size_t level = parser->level;
	4888	if (!mpack_tree_parse_node(tree, node))
	4889	return false;
	4890	--parser->stack[level].left;
	4891	++parser->stack[level].child;
	4892
	4893	mpack_assert(mpack_tree_error(tree) == mpack_ok,
	4894	"mpack_tree_parse_node() should have returned false due to error!");
	4895
	4896	// pop empty stack levels, exiting the outer loop when the stack is empty.
	4897	// (we could tail-optimize containers by pre-emptively popping empty
	4898	// stack levels before reading the new element, this way we wouldn't
	4899	// have to loop. but we eventually want to use the parse stack to give
	4900	// better error messages that contain the location of the error, so
	4901	// it needs to be complete.)
	4902	while (parser->stack[parser->level].left == 0) {
	4903	if (parser->level == 0)
	4904	return true;
	4905	--parser->level;
	4906	}
	4907	}
	4908	}
	4909
	4910	static void mpack_tree_cleanup(mpack_tree_t* tree) {
	4911	MPACK_UNUSED(tree);
	4912
	4913	#ifdef MPACK_MALLOC
	4914	if (tree->parser.stack_owned) {
	4915	MPACK_FREE(tree->parser.stack);
	4916	tree->parser.stack = NULL;
	4917	tree->parser.stack_owned = false;
	4918	}
	4919
	4920	mpack_tree_page_t* page = tree->next;
	4921	while (page != NULL) {
	4922	mpack_tree_page_t* next = page->next;
	4923	mpack_log("freeing page %p\n", page);
	4924	MPACK_FREE(page);
	4925	page = next;
	4926	}
	4927	tree->next = NULL;
	4928	#endif
	4929	}
	4930
	4931	static bool mpack_tree_parse_start(mpack_tree_t* tree) {
	4932	if (mpack_tree_error(tree) != mpack_ok)
	4933	return false;
	4934
	4935	mpack_tree_parser_t* parser = &tree->parser;
	4936	mpack_assert(parser->state != mpack_tree_parse_state_in_progress,
	4937	"previous parsing was not finished!");
	4938
	4939	if (parser->state == mpack_tree_parse_state_parsed)
	4940	mpack_tree_cleanup(tree);
	4941
	4942	mpack_log("starting parse\n");
	4943	tree->parser.state = mpack_tree_parse_state_in_progress;
	4944	tree->parser.current_node_reserved = 0;
	4945
	4946	// check if we previously parsed a tree
	4947	if (tree->size > 0) {
	4948	#ifdef MPACK_MALLOC
	4949	// if we're buffered, move the remaining data back to the
	4950	// start of the buffer
	4951	// TODO: This is not ideal performance-wise. We should only move data
	4952	// when we need to call the fill function.
	4953	// TODO: We could consider shrinking the buffer here, especially if we
	4954	// determine that the fill function is providing less than a quarter of
	4955	// the buffer size or if messages take up less than a quarter of the
	4956	// buffer size. Maybe this should be configurable.
	4957	if (tree->buffer != NULL) {
	4958	mpack_memmove(tree->buffer, tree->buffer + tree->size, tree->data_length - tree->size);
	4959	}
	4960	else
	4961	#endif
	4962	// otherwise advance past the parsed data
	4963	{
	4964	tree->data += tree->size;
	4965	}
	4966	tree->data_length -= tree->size;
	4967	tree->size = 0;
	4968	tree->node_count = 0;
	4969	}
	4970
	4971	// make sure we have at least one byte available before allocating anything
	4972	parser->possible_nodes_left = tree->data_length;
	4973	if (!mpack_tree_reserve_bytes(tree, sizeof(uint8_t))) {
	4974	tree->parser.state = mpack_tree_parse_state_not_started;
	4975	return false;
	4976	}
	4977	mpack_log("parsing tree at %p starting with byte %x\n", tree->data, (uint8_t)tree->data[0]);
	4978	parser->possible_nodes_left -= 1;
	4979	tree->node_count = 1;
	4980
	4981	#ifdef MPACK_MALLOC
	4982	parser->stack = parser->stack_local;
	4983	parser->stack_owned = false;
	4984	parser->stack_capacity = sizeof(parser->stack_local) / sizeof(*parser->stack_local);
	4985
	4986	if (tree->pool == NULL) {
	4987
	4988	// allocate first page
	4989	mpack_tree_page_t* page = (mpack_tree_page_t*)MPACK_MALLOC(MPACK_PAGE_ALLOC_SIZE);
	4990	mpack_log("allocated initial page %p of size %i count %i\n",
	4991	page, (int)MPACK_PAGE_ALLOC_SIZE, (int)MPACK_NODES_PER_PAGE);
	4992	if (page == NULL) {
	4993	tree->error = mpack_error_memory;
	4994	return false;
	4995	}
	4996	page->next = NULL;
	4997	tree->next = page;
	4998
	4999	parser->nodes = page->nodes;
	5000	parser->nodes_left = MPACK_NODES_PER_PAGE;
	5001	}
	5002	else
	5003	#endif
	5004	{
	5005	// otherwise use the provided pool
	5006	mpack_assert(tree->pool != NULL, "no pool provided?");
	5007	parser->nodes = tree->pool;
	5008	parser->nodes_left = tree->pool_count;
	5009	}
	5010
	5011	tree->root = parser->nodes;
	5012	++parser->nodes;
	5013	--parser->nodes_left;
	5014
	5015	parser->level = 0;
	5016	parser->stack[0].child = tree->root;
	5017	parser->stack[0].left = 1;
	5018
	5019	return true;
	5020	}
	5021
	5022	void mpack_tree_parse(mpack_tree_t* tree) {
	5023	if (mpack_tree_error(tree) != mpack_ok)
	5024	return;
	5025
	5026	if (tree->parser.state != mpack_tree_parse_state_in_progress) {
	5027	if (!mpack_tree_parse_start(tree)) {
	5028	mpack_tree_flag_error(tree, (tree->read_fn == NULL) ?
	5029	mpack_error_invalid : mpack_error_io);
	5030	return;
	5031	}
	5032	}
	5033
	5034	if (!mpack_tree_continue_parsing(tree)) {
	5035	if (mpack_tree_error(tree) != mpack_ok)
	5036	return;
	5037
	5038	// We're parsing synchronously on a blocking fill function. If we
	5039	// didn't completely finish parsing the tree, it's an error.
	5040	mpack_log("tree parsing incomplete. flagging error.\n");
	5041	mpack_tree_flag_error(tree, (tree->read_fn == NULL) ?
	5042	mpack_error_invalid : mpack_error_io);
	5043	return;
	5044	}
	5045
	5046	mpack_assert(mpack_tree_error(tree) == mpack_ok);
	5047	mpack_assert(tree->parser.level == 0);
	5048	tree->parser.state = mpack_tree_parse_state_parsed;
	5049	mpack_log("parsed tree of %i bytes, %i bytes left\n", (int)tree->size, (int)tree->parser.possible_nodes_left);
	5050	mpack_log("%i nodes in final page\n", (int)tree->parser.nodes_left);
	5051	}
	5052
	5053	bool mpack_tree_try_parse(mpack_tree_t* tree) {
	5054	if (mpack_tree_error(tree) != mpack_ok)
	5055	return false;
	5056
	5057	if (tree->parser.state != mpack_tree_parse_state_in_progress)
	5058	if (!mpack_tree_parse_start(tree))
	5059	return false;
	5060
	5061	if (!mpack_tree_continue_parsing(tree))
	5062	return false;
	5063
	5064	mpack_assert(mpack_tree_error(tree) == mpack_ok);
	5065	mpack_assert(tree->parser.level == 0);
	5066	tree->parser.state = mpack_tree_parse_state_parsed;
	5067	return true;
	5068	}
	5069
	5070
	5071
	5072	/*
	5073	* Tree functions
	5074	*/
	5075
	5076	mpack_node_t mpack_tree_root(mpack_tree_t* tree) {
	5077	if (mpack_tree_error(tree) != mpack_ok)
	5078	return mpack_tree_nil_node(tree);
	5079
	5080	// We check that a tree was parsed successfully and assert if not. You must
	5081	// call mpack_tree_parse() (or mpack_tree_try_parse() with a success
	5082	// result) in order to access the root node.
	5083	if (tree->parser.state != mpack_tree_parse_state_parsed) {
	5084	mpack_break("Tree has not been parsed! "
	5085	"Did you call mpack_tree_parse() or mpack_tree_try_parse()?");
	5086	mpack_tree_flag_error(tree, mpack_error_bug);
	5087	return mpack_tree_nil_node(tree);
	5088	}
	5089
	5090	return mpack_node(tree, tree->root);
	5091	}
	5092
	5093	static void mpack_tree_init_clear(mpack_tree_t* tree) {
	5094	mpack_memset(tree, 0, sizeof(*tree));
	5095	tree->nil_node.type = mpack_type_nil;
	5096	tree->missing_node.type = mpack_type_missing;
	5097	tree->max_size = SIZE_MAX;
	5098	tree->max_nodes = SIZE_MAX;
	5099	}
	5100
	5101	#ifdef MPACK_MALLOC
	5102	void mpack_tree_init_data(mpack_tree_t* tree, const char* data, size_t length) {
	5103	mpack_tree_init_clear(tree);
	5104
	5105	MPACK_STATIC_ASSERT(MPACK_NODE_PAGE_SIZE >= sizeof(mpack_tree_page_t),
	5106	"MPACK_NODE_PAGE_SIZE is too small");
	5107
	5108	MPACK_STATIC_ASSERT(MPACK_PAGE_ALLOC_SIZE <= MPACK_NODE_PAGE_SIZE,
	5109	"incorrect page rounding?");
	5110
	5111	tree->data = data;
	5112	tree->data_length = length;
	5113	tree->pool = NULL;
	5114	tree->pool_count = 0;
	5115	tree->next = NULL;
	5116
	5117	mpack_log("===========================\n");
	5118	mpack_log("initializing tree with data of size %i\n", (int)length);
	5119	}
	5120	#endif
	5121
	5122	void mpack_tree_init_pool(mpack_tree_t* tree, const char* data, size_t length,
	5123	mpack_node_data_t* node_pool, size_t node_pool_count)
	5124	{
	5125	mpack_tree_init_clear(tree);
	5126	#ifdef MPACK_MALLOC
	5127	tree->next = NULL;
	5128	#endif
	5129
	5130	if (node_pool_count == 0) {
	5131	mpack_break("initial page has no nodes!");
	5132	mpack_tree_flag_error(tree, mpack_error_bug);
	5133	return;
	5134	}
	5135
	5136	tree->data = data;
	5137	tree->data_length = length;
	5138	tree->pool = node_pool;
	5139	tree->pool_count = node_pool_count;
	5140
	5141	mpack_log("===========================\n");
	5142	mpack_log("initializing tree with data of size %i and pool of count %i\n",
	5143	(int)length, (int)node_pool_count);
	5144	}
	5145
	5146	void mpack_tree_init_error(mpack_tree_t* tree, mpack_error_t error) {
	5147	mpack_tree_init_clear(tree);
	5148	tree->error = error;
	5149
	5150	mpack_log("===========================\n");
	5151	mpack_log("initializing tree error state %i\n", (int)error);
	5152	}
	5153
	5154	#ifdef MPACK_MALLOC
	5155	void mpack_tree_init_stream(mpack_tree_t* tree, mpack_tree_read_t read_fn, void* context,
	5156	size_t max_message_size, size_t max_message_nodes) {
	5157	mpack_tree_init_clear(tree);
	5158
	5159	tree->read_fn = read_fn;
	5160	tree->context = context;
	5161
	5162	mpack_tree_set_limits(tree, max_message_size, max_message_nodes);
	5163	tree->max_size = max_message_size;
	5164	tree->max_nodes = max_message_nodes;
	5165
	5166	mpack_log("===========================\n");
	5167	mpack_log("initializing tree with stream, max size %i max nodes %i\n",
	5168	(int)max_message_size, (int)max_message_nodes);
	5169	}
	5170	#endif
	5171
	5172	void mpack_tree_set_limits(mpack_tree_t* tree, size_t max_message_size, size_t max_message_nodes) {
	5173	mpack_assert(max_message_size > 0);
	5174	mpack_assert(max_message_nodes > 0);
	5175	tree->max_size = max_message_size;
	5176	tree->max_nodes = max_message_nodes;
	5177	}
	5178
	5179	#if MPACK_STDIO
	5180	typedef struct mpack_file_tree_t {
	5181	char* data;
	5182	size_t size;
	5183	char buffer[MPACK_BUFFER_SIZE];
	5184	} mpack_file_tree_t;
	5185
	5186	static void mpack_file_tree_teardown(mpack_tree_t* tree) {
	5187	mpack_file_tree_t* file_tree = (mpack_file_tree_t*)tree->context;
	5188	MPACK_FREE(file_tree->data);
	5189	MPACK_FREE(file_tree);
	5190	}
	5191
	5192	static bool mpack_file_tree_read(mpack_tree_t* tree, mpack_file_tree_t* file_tree, FILE* file, size_t max_bytes) {
	5193
	5194	// get the file size
	5195	errno = 0;
	5196	int error = 0;
	5197	fseek(file, 0, SEEK_END);
	5198	error \|= errno;
	5199	long size = ftell(file);
	5200	error \|= errno;
	5201	fseek(file, 0, SEEK_SET);
	5202	error \|= errno;
	5203
	5204	// check for errors
	5205	if (error != 0 \|\| size < 0) {
	5206	mpack_tree_init_error(tree, mpack_error_io);
	5207	return false;
	5208	}
	5209	if (size == 0) {
	5210	mpack_tree_init_error(tree, mpack_error_invalid);
	5211	return false;
	5212	}
	5213
	5214	// make sure the size is less than max_bytes
	5215	// (this mess exists to safely convert between long and size_t regardless of their widths)
	5216	if (max_bytes != 0 && (((uint64_t)LONG_MAX > (uint64_t)SIZE_MAX && size > (long)SIZE_MAX) \|\| (size_t)size > max_bytes)) {
	5217	mpack_tree_init_error(tree, mpack_error_too_big);
	5218	return false;
	5219	}
	5220
	5221	// allocate data
	5222	file_tree->data = (char*)MPACK_MALLOC((size_t)size);
	5223	if (file_tree->data == NULL) {
	5224	mpack_tree_init_error(tree, mpack_error_memory);
	5225	return false;
	5226	}
	5227
	5228	// read the file
	5229	long total = 0;
	5230	while (total < size) {
	5231	size_t read = fread(file_tree->data + total, 1, (size_t)(size - total), file);
	5232	if (read <= 0) {
	5233	mpack_tree_init_error(tree, mpack_error_io);
	5234	MPACK_FREE(file_tree->data);
	5235	return false;
	5236	}
	5237	total += (long)read;
	5238	}
	5239
	5240	file_tree->size = (size_t)size;
	5241	return true;
	5242	}
	5243
	5244	static bool mpack_tree_file_check_max_bytes(mpack_tree_t* tree, size_t max_bytes) {
	5245
	5246	// the C STDIO family of file functions use long (e.g. ftell)
	5247	if (max_bytes > LONG_MAX) {
	5248	mpack_break("max_bytes of %" PRIu64 " is invalid, maximum is LONG_MAX", (uint64_t)max_bytes);
	5249	mpack_tree_init_error(tree, mpack_error_bug);
	5250	return false;
	5251	}
	5252
	5253	return true;
	5254	}
	5255
	5256	static void mpack_tree_init_stdfile_noclose(mpack_tree_t* tree, FILE* stdfile, size_t max_bytes) {
	5257
	5258	// allocate file tree
	5259	mpack_file_tree_t* file_tree = (mpack_file_tree_t*) MPACK_MALLOC(sizeof(mpack_file_tree_t));
	5260	if (file_tree == NULL) {
	5261	mpack_tree_init_error(tree, mpack_error_memory);
	5262	return;
	5263	}
	5264
	5265	// read all data
	5266	if (!mpack_file_tree_read(tree, file_tree, stdfile, max_bytes)) {
	5267	MPACK_FREE(file_tree);
	5268	return;
	5269	}
	5270
	5271	mpack_tree_init_data(tree, file_tree->data, file_tree->size);
	5272	mpack_tree_set_context(tree, file_tree);
	5273	mpack_tree_set_teardown(tree, mpack_file_tree_teardown);
	5274	}
	5275
	5276	void mpack_tree_init_stdfile(mpack_tree_t* tree, FILE* stdfile, size_t max_bytes, bool close_when_done) {
	5277	if (!mpack_tree_file_check_max_bytes(tree, max_bytes))
	5278	return;
	5279
	5280	mpack_tree_init_stdfile_noclose(tree, stdfile, max_bytes);
	5281
	5282	if (close_when_done)
	5283	fclose(stdfile);
	5284	}
	5285
	5286	void mpack_tree_init_filename(mpack_tree_t* tree, const char* filename, size_t max_bytes) {
	5287	if (!mpack_tree_file_check_max_bytes(tree, max_bytes))
	5288	return;
	5289
	5290	// open the file
	5291	FILE* file = fopen(filename, "rb");
	5292	if (!file) {
	5293	mpack_tree_init_error(tree, mpack_error_io);
	5294	return;
	5295	}
	5296
	5297	mpack_tree_init_stdfile(tree, file, max_bytes, true);
	5298	}
	5299	#endif
	5300
	5301	mpack_error_t mpack_tree_destroy(mpack_tree_t* tree) {
	5302	mpack_tree_cleanup(tree);
	5303
	5304	#ifdef MPACK_MALLOC
	5305	if (tree->buffer)
	5306	MPACK_FREE(tree->buffer);
	5307	#endif
	5308
	5309	if (tree->teardown)
	5310	tree->teardown(tree);
	5311	tree->teardown = NULL;
	5312
	5313	return tree->error;
	5314	}
	5315
	5316	void mpack_tree_flag_error(mpack_tree_t* tree, mpack_error_t error) {
	5317	if (tree->error == mpack_ok) {
	5318	mpack_log("tree %p setting error %i: %s\n", tree, (int)error, mpack_error_to_string(error));
	5319	tree->error = error;
	5320	if (tree->error_fn)
	5321	tree->error_fn(tree, error);
	5322	}
	5323
	5324	}
	5325
	5326
	5327
	5328	/*
	5329	* Node misc functions
	5330	*/
	5331
	5332	void mpack_node_flag_error(mpack_node_t node, mpack_error_t error) {
	5333	mpack_tree_flag_error(node.tree, error);
	5334	}
	5335
	5336	mpack_tag_t mpack_node_tag(mpack_node_t node) {
	5337	if (mpack_node_error(node) != mpack_ok)
	5338	return mpack_tag_nil();
	5339
	5340	mpack_tag_t tag = MPACK_TAG_ZERO;
	5341
	5342	tag.type = node.data->type;
	5343	switch (node.data->type) {
	5344	case mpack_type_missing:
	5345	// If a node is missing, I don't know if it makes sense to ask for
	5346	// a tag for it. We'll return a missing tag to match the missing
	5347	// node I guess, but attempting to use the tag for anything (like
	5348	// writing it for example) will flag mpack_error_bug.
	5349	break;
	5350	case mpack_type_nil: break;
	5351	case mpack_type_bool: tag.v.b = node.data->value.b; break;
	5352	case mpack_type_float: tag.v.f = node.data->value.f; break;
	5353	case mpack_type_double: tag.v.d = node.data->value.d; break;
	5354	case mpack_type_int: tag.v.i = node.data->value.i; break;
	5355	case mpack_type_uint: tag.v.u = node.data->value.u; break;
	5356
	5357	case mpack_type_str: tag.v.l = node.data->len; break;
	5358	case mpack_type_bin: tag.v.l = node.data->len; break;
	5359
	5360	#if MPACK_EXTENSIONS
	5361	case mpack_type_ext:
	5362	tag.v.l = node.data->len;
	5363	tag.exttype = mpack_node_exttype_unchecked(node);
	5364	break;
	5365	#endif
	5366
	5367	case mpack_type_array: tag.v.n = node.data->len; break;
	5368	case mpack_type_map: tag.v.n = node.data->len; break;
	5369
	5370	default:
	5371	mpack_assert(0, "unrecognized type %i", (int)node.data->type);
	5372	break;
	5373	}
	5374	return tag;
	5375	}
	5376
	5377	#if MPACK_DEBUG && MPACK_STDIO
	5378	static void mpack_node_print_element(mpack_node_t node, mpack_print_t* print, size_t depth) {
	5379	mpack_node_data_t* data = node.data;
	5380	switch (data->type) {
	5381	case mpack_type_str:
	5382	{
	5383	mpack_print_append_cstr(print, "\"");
	5384	const char* bytes = mpack_node_data_unchecked(node);
	5385	for (size_t i = 0; i < data->len; ++i) {
	5386	char c = bytes[i];
	5387	switch (c) {
	5388	case '\n': mpack_print_append_cstr(print, "\\n"); break;
	5389	case '\\': mpack_print_append_cstr(print, "\\\\"); break;
	5390	case '"': mpack_print_append_cstr(print, "\\\""); break;
	5391	default: mpack_print_append(print, &c, 1); break;
	5392	}
	5393	}
	5394	mpack_print_append_cstr(print, "\"");
	5395	}
	5396	break;
	5397
	5398	case mpack_type_array:
	5399	mpack_print_append_cstr(print, "[\n");
	5400	for (size_t i = 0; i < data->len; ++i) {
	5401	for (size_t j = 0; j < depth + 1; ++j)
	5402	mpack_print_append_cstr(print, " ");
	5403	mpack_node_print_element(mpack_node_array_at(node, i), print, depth + 1);
	5404	if (i != data->len - 1)
	5405	mpack_print_append_cstr(print, ",");
	5406	mpack_print_append_cstr(print, "\n");
	5407	}
	5408	for (size_t i = 0; i < depth; ++i)
	5409	mpack_print_append_cstr(print, " ");
	5410	mpack_print_append_cstr(print, "]");
	5411	break;
	5412
	5413	case mpack_type_map:
	5414	mpack_print_append_cstr(print, "{\n");
	5415	for (size_t i = 0; i < data->len; ++i) {
	5416	for (size_t j = 0; j < depth + 1; ++j)
	5417	mpack_print_append_cstr(print, " ");
	5418	mpack_node_print_element(mpack_node_map_key_at(node, i), print, depth + 1);
	5419	mpack_print_append_cstr(print, ": ");
	5420	mpack_node_print_element(mpack_node_map_value_at(node, i), print, depth + 1);
	5421	if (i != data->len - 1)
	5422	mpack_print_append_cstr(print, ",");
	5423	mpack_print_append_cstr(print, "\n");
	5424	}
	5425	for (size_t i = 0; i < depth; ++i)
	5426	mpack_print_append_cstr(print, " ");
	5427	mpack_print_append_cstr(print, "}");
	5428	break;
	5429
	5430	default:
	5431	{
	5432	const char* prefix = NULL;
	5433	size_t prefix_length = 0;
	5434	if (mpack_node_type(node) == mpack_type_bin
	5435	#if MPACK_EXTENSIONS
	5436	\|\| mpack_node_type(node) == mpack_type_ext
	5437	#endif
	5438	) {
	5439	prefix = mpack_node_data(node);
	5440	prefix_length = mpack_node_data_len(node);
	5441	}
	5442
	5443	char buf[256];
	5444	mpack_tag_t tag = mpack_node_tag(node);
	5445	mpack_tag_debug_pseudo_json(tag, buf, sizeof(buf), prefix, prefix_length);
	5446	mpack_print_append_cstr(print, buf);
	5447	}
	5448	break;
	5449	}
	5450	}
	5451
	5452	void mpack_node_print_to_buffer(mpack_node_t node, char* buffer, size_t buffer_size) {
	5453	if (buffer_size == 0) {
	5454	mpack_assert(false, "buffer size is zero!");
	5455	return;
	5456	}
	5457
	5458	mpack_print_t print;
	5459	mpack_memset(&print, 0, sizeof(print));
	5460	print.buffer = buffer;
	5461	print.size = buffer_size;
	5462	mpack_node_print_element(node, &print, 0);
	5463	mpack_print_append(&print, "", 1); // null-terminator
	5464	mpack_print_flush(&print);
	5465
	5466	// we always make sure there's a null-terminator at the end of the buffer
	5467	// in case we ran out of space.
	5468	print.buffer[print.size - 1] = '\0';
	5469	}
	5470
	5471	void mpack_node_print_to_callback(mpack_node_t node, mpack_print_callback_t callback, void* context) {
	5472	char buffer[1024];
	5473	mpack_print_t print;
	5474	mpack_memset(&print, 0, sizeof(print));
	5475	print.buffer = buffer;
	5476	print.size = sizeof(buffer);
	5477	print.callback = callback;
	5478	print.context = context;
	5479	mpack_node_print_element(node, &print, 0);
	5480	mpack_print_flush(&print);
	5481	}
	5482
	5483	void mpack_node_print_to_file(mpack_node_t node, FILE* file) {
	5484	mpack_assert(file != NULL, "file is NULL");
	5485
	5486	char buffer[1024];
	5487	mpack_print_t print;
	5488	mpack_memset(&print, 0, sizeof(print));
	5489	print.buffer = buffer;
	5490	print.size = sizeof(buffer);
	5491	print.callback = &mpack_print_file_callback;
	5492	print.context = file;
	5493
	5494	size_t depth = 2;
	5495	for (size_t i = 0; i < depth; ++i)
	5496	mpack_print_append_cstr(&print, " ");
	5497	mpack_node_print_element(node, &print, depth);
	5498	mpack_print_append_cstr(&print, "\n");
	5499	mpack_print_flush(&print);
	5500	}
	5501	#endif
	5502
	5503
	5504	/*
	5505	* Node Value Functions
	5506	*/
	5507
	5508	#if MPACK_EXTENSIONS
	5509	mpack_timestamp_t mpack_node_timestamp(mpack_node_t node) {
	5510	mpack_timestamp_t timestamp = {0, 0};
	5511
	5512	// we'll let mpack_node_exttype() do most checks
	5513	if (mpack_node_exttype(node) != MPACK_EXTTYPE_TIMESTAMP) {
	5514	mpack_log("exttype %i\n", mpack_node_exttype(node));
	5515	mpack_node_flag_error(node, mpack_error_type);
	5516	return timestamp;
	5517	}
	5518
	5519	const char* p = mpack_node_data_unchecked(node);
	5520
	5521	switch (node.data->len) {
	5522	case 4:
	5523	timestamp.nanoseconds = 0;
	5524	timestamp.seconds = mpack_load_u32(p);
	5525	break;
	5526
	5527	case 8: {
	5528	uint64_t value = mpack_load_u64(p);
	5529	timestamp.nanoseconds = (uint32_t)(value >> 34);
	5530	timestamp.seconds = value & ((UINT64_C(1) << 34) - 1);
	5531	break;
	5532	}
	5533
	5534	case 12:
	5535	timestamp.nanoseconds = mpack_load_u32(p);
	5536	timestamp.seconds = mpack_load_i64(p + 4);
	5537	break;
	5538
	5539	default:
	5540	mpack_tree_flag_error(node.tree, mpack_error_invalid);
	5541	return timestamp;
	5542	}
	5543
	5544	if (timestamp.nanoseconds > MPACK_TIMESTAMP_NANOSECONDS_MAX) {
	5545	mpack_tree_flag_error(node.tree, mpack_error_invalid);
	5546	mpack_timestamp_t zero = {0, 0};
	5547	return zero;
	5548	}
	5549
	5550	return timestamp;
	5551	}
	5552
	5553	int64_t mpack_node_timestamp_seconds(mpack_node_t node) {
	5554	return mpack_node_timestamp(node).seconds;
	5555	}
	5556
	5557	uint32_t mpack_node_timestamp_nanoseconds(mpack_node_t node) {
	5558	return mpack_node_timestamp(node).nanoseconds;
	5559	}
	5560	#endif
	5561
	5562
	5563
	5564	/*
	5565	* Node Data Functions
	5566	*/
	5567
	5568	void mpack_node_check_utf8(mpack_node_t node) {
	5569	if (mpack_node_error(node) != mpack_ok)
	5570	return;
	5571	mpack_node_data_t* data = node.data;
	5572	if (data->type != mpack_type_str \|\| !mpack_utf8_check(mpack_node_data_unchecked(node), data->len))
	5573	mpack_node_flag_error(node, mpack_error_type);
	5574	}
	5575
	5576	void mpack_node_check_utf8_cstr(mpack_node_t node) {
	5577	if (mpack_node_error(node) != mpack_ok)
	5578	return;
	5579	mpack_node_data_t* data = node.data;
	5580	if (data->type != mpack_type_str \|\| !mpack_utf8_check_no_null(mpack_node_data_unchecked(node), data->len))
	5581	mpack_node_flag_error(node, mpack_error_type);
	5582	}
	5583
	5584	size_t mpack_node_copy_data(mpack_node_t node, char* buffer, size_t bufsize) {
	5585	if (mpack_node_error(node) != mpack_ok)
	5586	return 0;
	5587
	5588	mpack_assert(bufsize == 0 \|\| buffer != NULL, "buffer is NULL for maximum of %i bytes", (int)bufsize);
	5589
	5590	mpack_type_t type = node.data->type;
	5591	if (type != mpack_type_str && type != mpack_type_bin
	5592	#if MPACK_EXTENSIONS
	5593	&& type != mpack_type_ext
	5594	#endif
	5595	) {
	5596	mpack_node_flag_error(node, mpack_error_type);
	5597	return 0;
	5598	}
	5599
	5600	if (node.data->len > bufsize) {
	5601	mpack_node_flag_error(node, mpack_error_too_big);
	5602	return 0;
	5603	}
	5604
	5605	mpack_memcpy(buffer, mpack_node_data_unchecked(node), node.data->len);
	5606	return (size_t)node.data->len;
	5607	}
	5608
	5609	size_t mpack_node_copy_utf8(mpack_node_t node, char* buffer, size_t bufsize) {
	5610	if (mpack_node_error(node) != mpack_ok)
	5611	return 0;
	5612
	5613	mpack_assert(bufsize == 0 \|\| buffer != NULL, "buffer is NULL for maximum of %i bytes", (int)bufsize);
	5614
	5615	mpack_type_t type = node.data->type;
	5616	if (type != mpack_type_str) {
	5617	mpack_node_flag_error(node, mpack_error_type);
	5618	return 0;
	5619	}
	5620
	5621	if (node.data->len > bufsize) {
	5622	mpack_node_flag_error(node, mpack_error_too_big);
	5623	return 0;
	5624	}
	5625
	5626	if (!mpack_utf8_check(mpack_node_data_unchecked(node), node.data->len)) {
	5627	mpack_node_flag_error(node, mpack_error_type);
	5628	return 0;
	5629	}
	5630
	5631	mpack_memcpy(buffer, mpack_node_data_unchecked(node), node.data->len);
	5632	return (size_t)node.data->len;
	5633	}
	5634
	5635	void mpack_node_copy_cstr(mpack_node_t node, char* buffer, size_t bufsize) {
	5636
	5637	// we can't break here because the error isn't recoverable; we
	5638	// have to add a null-terminator.
	5639	mpack_assert(buffer != NULL, "buffer is NULL");
	5640	mpack_assert(bufsize >= 1, "buffer size is zero; you must have room for at least a null-terminator");
	5641
	5642	if (mpack_node_error(node) != mpack_ok) {
	5643	buffer[0] = '\0';
	5644	return;
	5645	}
	5646
	5647	if (node.data->type != mpack_type_str) {
	5648	buffer[0] = '\0';
	5649	mpack_node_flag_error(node, mpack_error_type);
	5650	return;
	5651	}
	5652
	5653	if (node.data->len > bufsize - 1) {
	5654	buffer[0] = '\0';
	5655	mpack_node_flag_error(node, mpack_error_too_big);
	5656	return;
	5657	}
	5658
	5659	if (!mpack_str_check_no_null(mpack_node_data_unchecked(node), node.data->len)) {
	5660	buffer[0] = '\0';
	5661	mpack_node_flag_error(node, mpack_error_type);
	5662	return;
	5663	}
	5664
	5665	mpack_memcpy(buffer, mpack_node_data_unchecked(node), node.data->len);
	5666	buffer[node.data->len] = '\0';
	5667	}
	5668
	5669	void mpack_node_copy_utf8_cstr(mpack_node_t node, char* buffer, size_t bufsize) {
	5670
	5671	// we can't break here because the error isn't recoverable; we
	5672	// have to add a null-terminator.
	5673	mpack_assert(buffer != NULL, "buffer is NULL");
	5674	mpack_assert(bufsize >= 1, "buffer size is zero; you must have room for at least a null-terminator");
	5675
	5676	if (mpack_node_error(node) != mpack_ok) {
	5677	buffer[0] = '\0';
	5678	return;
	5679	}
	5680
	5681	if (node.data->type != mpack_type_str) {
	5682	buffer[0] = '\0';
	5683	mpack_node_flag_error(node, mpack_error_type);
	5684	return;
	5685	}
	5686
	5687	if (node.data->len > bufsize - 1) {
	5688	buffer[0] = '\0';
	5689	mpack_node_flag_error(node, mpack_error_too_big);
	5690	return;
	5691	}
	5692
	5693	if (!mpack_utf8_check_no_null(mpack_node_data_unchecked(node), node.data->len)) {
	5694	buffer[0] = '\0';
	5695	mpack_node_flag_error(node, mpack_error_type);
	5696	return;
	5697	}
	5698
	5699	mpack_memcpy(buffer, mpack_node_data_unchecked(node), node.data->len);
	5700	buffer[node.data->len] = '\0';
	5701	}
	5702
	5703	#ifdef MPACK_MALLOC
	5704	char* mpack_node_data_alloc(mpack_node_t node, size_t maxlen) {
	5705	if (mpack_node_error(node) != mpack_ok)
	5706	return NULL;
	5707
	5708	// make sure this is a valid data type
	5709	mpack_type_t type = node.data->type;
	5710	if (type != mpack_type_str && type != mpack_type_bin
	5711	#if MPACK_EXTENSIONS
	5712	&& type != mpack_type_ext
	5713	#endif
	5714	) {
	5715	mpack_node_flag_error(node, mpack_error_type);
	5716	return NULL;
	5717	}
	5718
	5719	if (node.data->len > maxlen) {
	5720	mpack_node_flag_error(node, mpack_error_too_big);
	5721	return NULL;
	5722	}
	5723
	5724	char* ret = (char*) MPACK_MALLOC((size_t)node.data->len);
	5725	if (ret == NULL) {
	5726	mpack_node_flag_error(node, mpack_error_memory);
	5727	return NULL;
	5728	}
	5729
	5730	mpack_memcpy(ret, mpack_node_data_unchecked(node), node.data->len);
	5731	return ret;
	5732	}
	5733
	5734	char* mpack_node_cstr_alloc(mpack_node_t node, size_t maxlen) {
	5735	if (mpack_node_error(node) != mpack_ok)
	5736	return NULL;
	5737
	5738	// make sure maxlen makes sense
	5739	if (maxlen < 1) {
	5740	mpack_break("maxlen is zero; you must have room for at least a null-terminator");
	5741	mpack_node_flag_error(node, mpack_error_bug);
	5742	return NULL;
	5743	}
	5744
	5745	if (node.data->type != mpack_type_str) {
	5746	mpack_node_flag_error(node, mpack_error_type);
	5747	return NULL;
	5748	}
	5749
	5750	if (node.data->len > maxlen - 1) {
	5751	mpack_node_flag_error(node, mpack_error_too_big);
	5752	return NULL;
	5753	}
	5754
	5755	if (!mpack_str_check_no_null(mpack_node_data_unchecked(node), node.data->len)) {
	5756	mpack_node_flag_error(node, mpack_error_type);
	5757	return NULL;
	5758	}
	5759
	5760	char* ret = (char*) MPACK_MALLOC((size_t)(node.data->len + 1));
	5761	if (ret == NULL) {
	5762	mpack_node_flag_error(node, mpack_error_memory);
	5763	return NULL;
	5764	}
	5765
	5766	mpack_memcpy(ret, mpack_node_data_unchecked(node), node.data->len);
	5767	ret[node.data->len] = '\0';
	5768	return ret;
	5769	}
	5770
	5771	char* mpack_node_utf8_cstr_alloc(mpack_node_t node, size_t maxlen) {
	5772	if (mpack_node_error(node) != mpack_ok)
	5773	return NULL;
	5774
	5775	// make sure maxlen makes sense
	5776	if (maxlen < 1) {
	5777	mpack_break("maxlen is zero; you must have room for at least a null-terminator");
	5778	mpack_node_flag_error(node, mpack_error_bug);
	5779	return NULL;
	5780	}
	5781
	5782	if (node.data->type != mpack_type_str) {
	5783	mpack_node_flag_error(node, mpack_error_type);
	5784	return NULL;
	5785	}
	5786
	5787	if (node.data->len > maxlen - 1) {
	5788	mpack_node_flag_error(node, mpack_error_too_big);
	5789	return NULL;
	5790	}
	5791
	5792	if (!mpack_utf8_check_no_null(mpack_node_data_unchecked(node), node.data->len)) {
	5793	mpack_node_flag_error(node, mpack_error_type);
	5794	return NULL;
	5795	}
	5796
	5797	char* ret = (char*) MPACK_MALLOC((size_t)(node.data->len + 1));
	5798	if (ret == NULL) {
	5799	mpack_node_flag_error(node, mpack_error_memory);
	5800	return NULL;
	5801	}
	5802
	5803	mpack_memcpy(ret, mpack_node_data_unchecked(node), node.data->len);
	5804	ret[node.data->len] = '\0';
	5805	return ret;
	5806	}
	5807	#endif
	5808
	5809
	5810	/*
	5811	* Compound Node Functions
	5812	*/
	5813
	5814	static mpack_node_data_t* mpack_node_map_int_impl(mpack_node_t node, int64_t num) {
	5815	if (mpack_node_error(node) != mpack_ok)
	5816	return NULL;
	5817
	5818	if (node.data->type != mpack_type_map) {
	5819	mpack_node_flag_error(node, mpack_error_type);
	5820	return NULL;
	5821	}
	5822
	5823	mpack_node_data_t* found = NULL;
	5824
	5825	for (size_t i = 0; i < node.data->len; ++i) {
	5826	mpack_node_data_t* key = mpack_node_child(node, i * 2);
	5827
	5828	if ((key->type == mpack_type_int && key->value.i == num) \|\|
	5829	(key->type == mpack_type_uint && num >= 0 && key->value.u == (uint64_t)num))
	5830	{
	5831	if (found) {
	5832	mpack_node_flag_error(node, mpack_error_data);
	5833	return NULL;
	5834	}
	5835	found = mpack_node_child(node, i * 2 + 1);
	5836	}
	5837	}
	5838
	5839	if (found)
	5840	return found;
	5841
	5842	return NULL;
	5843	}
	5844
	5845	static mpack_node_data_t* mpack_node_map_uint_impl(mpack_node_t node, uint64_t num) {
	5846	if (mpack_node_error(node) != mpack_ok)
	5847	return NULL;
	5848
	5849	if (node.data->type != mpack_type_map) {
	5850	mpack_node_flag_error(node, mpack_error_type);
	5851	return NULL;
	5852	}
	5853
	5854	mpack_node_data_t* found = NULL;
	5855
	5856	for (size_t i = 0; i < node.data->len; ++i) {
	5857	mpack_node_data_t* key = mpack_node_child(node, i * 2);
	5858
	5859	if ((key->type == mpack_type_uint && key->value.u == num) \|\|
	5860	(key->type == mpack_type_int && key->value.i >= 0 && (uint64_t)key->value.i == num))
	5861	{
	5862	if (found) {
	5863	mpack_node_flag_error(node, mpack_error_data);
	5864	return NULL;
	5865	}
	5866	found = mpack_node_child(node, i * 2 + 1);
	5867	}
	5868	}
	5869
	5870	if (found)
	5871	return found;
	5872
	5873	return NULL;
	5874	}
	5875
	5876	static mpack_node_data_t* mpack_node_map_str_impl(mpack_node_t node, const char* str, size_t length) {
	5877	if (mpack_node_error(node) != mpack_ok)
	5878	return NULL;
	5879
	5880	mpack_assert(length == 0 \|\| str != NULL, "str of length %i is NULL", (int)length);
	5881
	5882	if (node.data->type != mpack_type_map) {
	5883	mpack_node_flag_error(node, mpack_error_type);
	5884	return NULL;
	5885	}
	5886
	5887	mpack_tree_t* tree = node.tree;
	5888	mpack_node_data_t* found = NULL;
	5889
	5890	for (size_t i = 0; i < node.data->len; ++i) {
	5891	mpack_node_data_t* key = mpack_node_child(node, i * 2);
	5892
	5893	if (key->type == mpack_type_str && key->len == length &&
	5894	mpack_memcmp(str, mpack_node_data_unchecked(mpack_node(tree, key)), length) == 0) {
	5895	if (found) {
	5896	mpack_node_flag_error(node, mpack_error_data);
	5897	return NULL;
	5898	}
	5899	found = mpack_node_child(node, i * 2 + 1);
	5900	}
	5901	}
	5902
	5903	if (found)
	5904	return found;
	5905
	5906	return NULL;
	5907	}
	5908
	5909	static mpack_node_t mpack_node_wrap_lookup(mpack_tree_t* tree, mpack_node_data_t* data) {
	5910	if (!data) {
	5911	if (tree->error == mpack_ok)
	5912	mpack_tree_flag_error(tree, mpack_error_data);
	5913	return mpack_tree_nil_node(tree);
	5914	}
	5915	return mpack_node(tree, data);
	5916	}
	5917
	5918	static mpack_node_t mpack_node_wrap_lookup_optional(mpack_tree_t* tree, mpack_node_data_t* data) {
	5919	if (!data) {
	5920	if (tree->error == mpack_ok)
	5921	return mpack_tree_missing_node(tree);
	5922	return mpack_tree_nil_node(tree);
	5923	}
	5924	return mpack_node(tree, data);
	5925	}
	5926
	5927	mpack_node_t mpack_node_map_int(mpack_node_t node, int64_t num) {
	5928	return mpack_node_wrap_lookup(node.tree, mpack_node_map_int_impl(node, num));
	5929	}
	5930
	5931	mpack_node_t mpack_node_map_int_optional(mpack_node_t node, int64_t num) {
	5932	return mpack_node_wrap_lookup_optional(node.tree, mpack_node_map_int_impl(node, num));
	5933	}
	5934
	5935	mpack_node_t mpack_node_map_uint(mpack_node_t node, uint64_t num) {
	5936	return mpack_node_wrap_lookup(node.tree, mpack_node_map_uint_impl(node, num));
	5937	}
	5938
	5939	mpack_node_t mpack_node_map_uint_optional(mpack_node_t node, uint64_t num) {
	5940	return mpack_node_wrap_lookup_optional(node.tree, mpack_node_map_uint_impl(node, num));
	5941	}
	5942
	5943	mpack_node_t mpack_node_map_str(mpack_node_t node, const char* str, size_t length) {
	5944	return mpack_node_wrap_lookup(node.tree, mpack_node_map_str_impl(node, str, length));
	5945	}
	5946
	5947	mpack_node_t mpack_node_map_str_optional(mpack_node_t node, const char* str, size_t length) {
	5948	return mpack_node_wrap_lookup_optional(node.tree, mpack_node_map_str_impl(node, str, length));
	5949	}
	5950
	5951	mpack_node_t mpack_node_map_cstr(mpack_node_t node, const char* cstr) {
	5952	mpack_assert(cstr != NULL, "cstr is NULL");
	5953	return mpack_node_map_str(node, cstr, mpack_strlen(cstr));
	5954	}
	5955
	5956	mpack_node_t mpack_node_map_cstr_optional(mpack_node_t node, const char* cstr) {
	5957	mpack_assert(cstr != NULL, "cstr is NULL");
	5958	return mpack_node_map_str_optional(node, cstr, mpack_strlen(cstr));
	5959	}
	5960
	5961	bool mpack_node_map_contains_int(mpack_node_t node, int64_t num) {
	5962	return mpack_node_map_int_impl(node, num) != NULL;
	5963	}
	5964
	5965	bool mpack_node_map_contains_uint(mpack_node_t node, uint64_t num) {
	5966	return mpack_node_map_uint_impl(node, num) != NULL;
	5967	}
	5968
	5969	bool mpack_node_map_contains_str(mpack_node_t node, const char* str, size_t length) {
	5970	return mpack_node_map_str_impl(node, str, length) != NULL;
	5971	}
	5972
	5973	bool mpack_node_map_contains_cstr(mpack_node_t node, const char* cstr) {
	5974	mpack_assert(cstr != NULL, "cstr is NULL");
	5975	return mpack_node_map_contains_str(node, cstr, mpack_strlen(cstr));
	5976	}
	5977
	5978	size_t mpack_node_enum_optional(mpack_node_t node, const char* strings[], size_t count) {
	5979	if (mpack_node_error(node) != mpack_ok)
	5980	return count;
	5981
	5982	// the value is only recognized if it is a string
	5983	if (mpack_node_type(node) != mpack_type_str)
	5984	return count;
	5985
	5986	// fetch the string
	5987	const char* key = mpack_node_str(node);
	5988	size_t keylen = mpack_node_strlen(node);
	5989	mpack_assert(mpack_node_error(node) == mpack_ok, "these should not fail");
	5990
	5991	// find what key it matches
	5992	for (size_t i = 0; i < count; ++i) {
	5993	const char* other = strings[i];
	5994	size_t otherlen = mpack_strlen(other);
	5995	if (keylen == otherlen && mpack_memcmp(key, other, keylen) == 0)
	5996	return i;
	5997	}
	5998
	5999	// no matches
	6000	return count;
	6001	}
	6002
	6003	size_t mpack_node_enum(mpack_node_t node, const char* strings[], size_t count) {
	6004	size_t value = mpack_node_enum_optional(node, strings, count);
	6005	if (value == count)
	6006	mpack_node_flag_error(node, mpack_error_type);
	6007	return value;
	6008	}
	6009
	6010	mpack_type_t mpack_node_type(mpack_node_t node) {
	6011	if (mpack_node_error(node) != mpack_ok)
	6012	return mpack_type_nil;
	6013	return node.data->type;
	6014	}
	6015
	6016	bool mpack_node_is_nil(mpack_node_t node) {
	6017	if (mpack_node_error(node) != mpack_ok) {
	6018	// All nodes are treated as nil nodes when we are in error.
	6019	return true;
	6020	}
	6021	return node.data->type == mpack_type_nil;
	6022	}
	6023
	6024	bool mpack_node_is_missing(mpack_node_t node) {
	6025	if (mpack_node_error(node) != mpack_ok) {
	6026	// errors still return nil nodes, not missing nodes.
	6027	return false;
	6028	}
	6029	return node.data->type == mpack_type_missing;
	6030	}
	6031
	6032	void mpack_node_nil(mpack_node_t node) {
	6033	if (mpack_node_error(node) != mpack_ok)
	6034	return;
	6035	if (node.data->type != mpack_type_nil)
	6036	mpack_node_flag_error(node, mpack_error_type);
	6037	}
	6038
	6039	void mpack_node_missing(mpack_node_t node) {
	6040	if (mpack_node_error(node) != mpack_ok)
	6041	return;
	6042	if (node.data->type != mpack_type_missing)
	6043	mpack_node_flag_error(node, mpack_error_type);
	6044	}
	6045
	6046	bool mpack_node_bool(mpack_node_t node) {
	6047	if (mpack_node_error(node) != mpack_ok)
	6048	return false;
	6049
	6050	if (node.data->type == mpack_type_bool)
	6051	return node.data->value.b;
	6052
	6053	mpack_node_flag_error(node, mpack_error_type);
	6054	return false;
	6055	}
	6056
	6057	void mpack_node_true(mpack_node_t node) {
	6058	if (mpack_node_bool(node) != true)
	6059	mpack_node_flag_error(node, mpack_error_type);
	6060	}
	6061
	6062	void mpack_node_false(mpack_node_t node) {
	6063	if (mpack_node_bool(node) != false)
	6064	mpack_node_flag_error(node, mpack_error_type);
	6065	}
	6066
	6067	uint8_t mpack_node_u8(mpack_node_t node) {
	6068	if (mpack_node_error(node) != mpack_ok)
	6069	return 0;
	6070
	6071	if (node.data->type == mpack_type_uint) {
	6072	if (node.data->value.u <= UINT8_MAX)
	6073	return (uint8_t)node.data->value.u;
	6074	} else if (node.data->type == mpack_type_int) {
	6075	if (node.data->value.i >= 0 && node.data->value.i <= UINT8_MAX)
	6076	return (uint8_t)node.data->value.i;
	6077	}
	6078
	6079	mpack_node_flag_error(node, mpack_error_type);
	6080	return 0;
	6081	}
	6082
	6083	int8_t mpack_node_i8(mpack_node_t node) {
	6084	if (mpack_node_error(node) != mpack_ok)
	6085	return 0;
	6086
	6087	if (node.data->type == mpack_type_uint) {
	6088	if (node.data->value.u <= INT8_MAX)
	6089	return (int8_t)node.data->value.u;
	6090	} else if (node.data->type == mpack_type_int) {
	6091	if (node.data->value.i >= INT8_MIN && node.data->value.i <= INT8_MAX)
	6092	return (int8_t)node.data->value.i;
	6093	}
	6094
	6095	mpack_node_flag_error(node, mpack_error_type);
	6096	return 0;
	6097	}
	6098
	6099	uint16_t mpack_node_u16(mpack_node_t node) {
	6100	if (mpack_node_error(node) != mpack_ok)
	6101	return 0;
	6102
	6103	if (node.data->type == mpack_type_uint) {
	6104	if (node.data->value.u <= UINT16_MAX)
	6105	return (uint16_t)node.data->value.u;
	6106	} else if (node.data->type == mpack_type_int) {
	6107	if (node.data->value.i >= 0 && node.data->value.i <= UINT16_MAX)
	6108	return (uint16_t)node.data->value.i;
	6109	}
	6110
	6111	mpack_node_flag_error(node, mpack_error_type);
	6112	return 0;
	6113	}
	6114
	6115	int16_t mpack_node_i16(mpack_node_t node) {
	6116	if (mpack_node_error(node) != mpack_ok)
	6117	return 0;
	6118
	6119	if (node.data->type == mpack_type_uint) {
	6120	if (node.data->value.u <= INT16_MAX)
	6121	return (int16_t)node.data->value.u;
	6122	} else if (node.data->type == mpack_type_int) {
	6123	if (node.data->value.i >= INT16_MIN && node.data->value.i <= INT16_MAX)
	6124	return (int16_t)node.data->value.i;
	6125	}
	6126
	6127	mpack_node_flag_error(node, mpack_error_type);
	6128	return 0;
	6129	}
	6130
	6131	uint32_t mpack_node_u32(mpack_node_t node) {
	6132	if (mpack_node_error(node) != mpack_ok)
	6133	return 0;
	6134
	6135	if (node.data->type == mpack_type_uint) {
	6136	if (node.data->value.u <= UINT32_MAX)
	6137	return (uint32_t)node.data->value.u;
	6138	} else if (node.data->type == mpack_type_int) {
	6139	if (node.data->value.i >= 0 && node.data->value.i <= UINT32_MAX)
	6140	return (uint32_t)node.data->value.i;
	6141	}
	6142
	6143	mpack_node_flag_error(node, mpack_error_type);
	6144	return 0;
	6145	}
	6146
	6147	int32_t mpack_node_i32(mpack_node_t node) {
	6148	if (mpack_node_error(node) != mpack_ok)
	6149	return 0;
	6150
	6151	if (node.data->type == mpack_type_uint) {
	6152	if (node.data->value.u <= INT32_MAX)
	6153	return (int32_t)node.data->value.u;
	6154	} else if (node.data->type == mpack_type_int) {
	6155	if (node.data->value.i >= INT32_MIN && node.data->value.i <= INT32_MAX)
	6156	return (int32_t)node.data->value.i;
	6157	}
	6158
	6159	mpack_node_flag_error(node, mpack_error_type);
	6160	return 0;
	6161	}
	6162
	6163	uint64_t mpack_node_u64(mpack_node_t node) {
	6164	if (mpack_node_error(node) != mpack_ok)
	6165	return 0;
	6166
	6167	if (node.data->type == mpack_type_uint) {
	6168	return node.data->value.u;
	6169	} else if (node.data->type == mpack_type_int) {
	6170	if (node.data->value.i >= 0)
	6171	return (uint64_t)node.data->value.i;
	6172	}
	6173
	6174	mpack_node_flag_error(node, mpack_error_type);
	6175	return 0;
	6176	}
	6177
	6178	int64_t mpack_node_i64(mpack_node_t node) {
	6179	if (mpack_node_error(node) != mpack_ok)
	6180	return 0;
	6181
	6182	if (node.data->type == mpack_type_uint) {
	6183	if (node.data->value.u <= (uint64_t)INT64_MAX)
	6184	return (int64_t)node.data->value.u;
	6185	} else if (node.data->type == mpack_type_int) {
	6186	return node.data->value.i;
	6187	}
	6188
	6189	mpack_node_flag_error(node, mpack_error_type);
	6190	return 0;
	6191	}
	6192
	6193	unsigned int mpack_node_uint(mpack_node_t node) {
	6194
	6195	// This should be true at compile-time, so this just wraps the 32-bit function.
	6196	if (sizeof(unsigned int) == 4)
	6197	return (unsigned int)mpack_node_u32(node);
	6198
	6199	// Otherwise we use u64 and check the range.
	6200	uint64_t val = mpack_node_u64(node);
	6201	if (val <= UINT_MAX)
	6202	return (unsigned int)val;
	6203
	6204	mpack_node_flag_error(node, mpack_error_type);
	6205	return 0;
	6206	}
	6207
	6208	int mpack_node_int(mpack_node_t node) {
	6209
	6210	// This should be true at compile-time, so this just wraps the 32-bit function.
	6211	if (sizeof(int) == 4)
	6212	return (int)mpack_node_i32(node);
	6213
	6214	// Otherwise we use i64 and check the range.
	6215	int64_t val = mpack_node_i64(node);
	6216	if (val >= INT_MIN && val <= INT_MAX)
	6217	return (int)val;
	6218
	6219	mpack_node_flag_error(node, mpack_error_type);
	6220	return 0;
	6221	}
	6222
	6223	float mpack_node_float(mpack_node_t node) {
	6224	if (mpack_node_error(node) != mpack_ok)
	6225	return 0.0f;
	6226
	6227	if (node.data->type == mpack_type_uint)
	6228	return (float)node.data->value.u;
	6229	else if (node.data->type == mpack_type_int)
	6230	return (float)node.data->value.i;
	6231	else if (node.data->type == mpack_type_float)
	6232	return node.data->value.f;
	6233	else if (node.data->type == mpack_type_double)
	6234	return (float)node.data->value.d;
	6235
	6236	mpack_node_flag_error(node, mpack_error_type);
	6237	return 0.0f;
	6238	}
	6239
	6240	double mpack_node_double(mpack_node_t node) {
	6241	if (mpack_node_error(node) != mpack_ok)
	6242	return 0.0;
	6243
	6244	if (node.data->type == mpack_type_uint)
	6245	return (double)node.data->value.u;
	6246	else if (node.data->type == mpack_type_int)
	6247	return (double)node.data->value.i;
	6248	else if (node.data->type == mpack_type_float)
	6249	return (double)node.data->value.f;
	6250	else if (node.data->type == mpack_type_double)
	6251	return node.data->value.d;
	6252
	6253	mpack_node_flag_error(node, mpack_error_type);
	6254	return 0.0;
	6255	}
	6256
	6257	float mpack_node_float_strict(mpack_node_t node) {
	6258	if (mpack_node_error(node) != mpack_ok)
	6259	return 0.0f;
	6260
	6261	if (node.data->type == mpack_type_float)
	6262	return node.data->value.f;
	6263
	6264	mpack_node_flag_error(node, mpack_error_type);
	6265	return 0.0f;
	6266	}
	6267
	6268	double mpack_node_double_strict(mpack_node_t node) {
	6269	if (mpack_node_error(node) != mpack_ok)
	6270	return 0.0;
	6271
	6272	if (node.data->type == mpack_type_float)
	6273	return (double)node.data->value.f;
	6274	else if (node.data->type == mpack_type_double)
	6275	return node.data->value.d;
	6276
	6277	mpack_node_flag_error(node, mpack_error_type);
	6278	return 0.0;
	6279	}
	6280
	6281	#if MPACK_EXTENSIONS
	6282	int8_t mpack_node_exttype(mpack_node_t node) {
	6283	if (mpack_node_error(node) != mpack_ok)
	6284	return 0;
	6285
	6286	if (node.data->type == mpack_type_ext)
	6287	return mpack_node_exttype_unchecked(node);
	6288
	6289	mpack_node_flag_error(node, mpack_error_type);
	6290	return 0;
	6291	}
	6292	#endif
	6293
	6294	uint32_t mpack_node_data_len(mpack_node_t node) {
	6295	if (mpack_node_error(node) != mpack_ok)
	6296	return 0;
	6297
	6298	mpack_type_t type = node.data->type;
	6299	if (type == mpack_type_str \|\| type == mpack_type_bin
	6300	#if MPACK_EXTENSIONS
	6301	\|\| type == mpack_type_ext
	6302	#endif
	6303	)
	6304	return (uint32_t)node.data->len;
	6305
	6306	mpack_node_flag_error(node, mpack_error_type);
	6307	return 0;
	6308	}
	6309
	6310	size_t mpack_node_strlen(mpack_node_t node) {
	6311	if (mpack_node_error(node) != mpack_ok)
	6312	return 0;
	6313
	6314	if (node.data->type == mpack_type_str)
	6315	return (size_t)node.data->len;
	6316
	6317	mpack_node_flag_error(node, mpack_error_type);
	6318	return 0;
	6319	}
	6320
	6321	const char* mpack_node_str(mpack_node_t node) {
	6322	if (mpack_node_error(node) != mpack_ok)
	6323	return NULL;
	6324
	6325	mpack_type_t type = node.data->type;
	6326	if (type == mpack_type_str)
	6327	return mpack_node_data_unchecked(node);
	6328
	6329	mpack_node_flag_error(node, mpack_error_type);
	6330	return NULL;
	6331	}
	6332
	6333	const char* mpack_node_data(mpack_node_t node) {
	6334	if (mpack_node_error(node) != mpack_ok)
	6335	return NULL;
	6336
	6337	mpack_type_t type = node.data->type;
	6338	if (type == mpack_type_str \|\| type == mpack_type_bin
	6339	#if MPACK_EXTENSIONS
	6340	\|\| type == mpack_type_ext
	6341	#endif
	6342	)
	6343	return mpack_node_data_unchecked(node);
	6344
	6345	mpack_node_flag_error(node, mpack_error_type);
	6346	return NULL;
	6347	}
	6348
	6349	const char* mpack_node_bin_data(mpack_node_t node) {
	6350	if (mpack_node_error(node) != mpack_ok)
	6351	return NULL;
	6352
	6353	if (node.data->type == mpack_type_bin)
	6354	return mpack_node_data_unchecked(node);
	6355
	6356	mpack_node_flag_error(node, mpack_error_type);
	6357	return NULL;
	6358	}
	6359
	6360	size_t mpack_node_bin_size(mpack_node_t node) {
	6361	if (mpack_node_error(node) != mpack_ok)
	6362	return 0;
	6363
	6364	if (node.data->type == mpack_type_bin)
	6365	return (size_t)node.data->len;
	6366
	6367	mpack_node_flag_error(node, mpack_error_type);
	6368	return 0;
	6369	}
	6370
	6371	size_t mpack_node_array_length(mpack_node_t node) {
	6372	if (mpack_node_error(node) != mpack_ok)
	6373	return 0;
	6374
	6375	if (node.data->type != mpack_type_array) {
	6376	mpack_node_flag_error(node, mpack_error_type);
	6377	return 0;
	6378	}
	6379
	6380	return (size_t)node.data->len;
	6381	}
	6382
	6383	mpack_node_t mpack_node_array_at(mpack_node_t node, size_t index) {
	6384	if (mpack_node_error(node) != mpack_ok)
	6385	return mpack_tree_nil_node(node.tree);
	6386
	6387	if (node.data->type != mpack_type_array) {
	6388	mpack_node_flag_error(node, mpack_error_type);
	6389	return mpack_tree_nil_node(node.tree);
	6390	}
	6391
	6392	if (index >= node.data->len) {
	6393	mpack_node_flag_error(node, mpack_error_data);
	6394	return mpack_tree_nil_node(node.tree);
	6395	}
	6396
	6397	return mpack_node(node.tree, mpack_node_child(node, index));
	6398	}
	6399
	6400	size_t mpack_node_map_count(mpack_node_t node) {
	6401	if (mpack_node_error(node) != mpack_ok)
	6402	return 0;
	6403
	6404	if (node.data->type != mpack_type_map) {
	6405	mpack_node_flag_error(node, mpack_error_type);
	6406	return 0;
	6407	}
	6408
	6409	return node.data->len;
	6410	}
	6411
	6412	// internal node map lookup
	6413	static mpack_node_t mpack_node_map_at(mpack_node_t node, size_t index, size_t offset) {
	6414	if (mpack_node_error(node) != mpack_ok)
	6415	return mpack_tree_nil_node(node.tree);
	6416
	6417	if (node.data->type != mpack_type_map) {
	6418	mpack_node_flag_error(node, mpack_error_type);
	6419	return mpack_tree_nil_node(node.tree);
	6420	}
	6421
	6422	if (index >= node.data->len) {
	6423	mpack_node_flag_error(node, mpack_error_data);
	6424	return mpack_tree_nil_node(node.tree);
	6425	}
	6426
	6427	return mpack_node(node.tree, mpack_node_child(node, index * 2 + offset));
	6428	}
	6429
	6430	mpack_node_t mpack_node_map_key_at(mpack_node_t node, size_t index) {
	6431	return mpack_node_map_at(node, index, 0);
	6432	}
	6433
	6434	mpack_node_t mpack_node_map_value_at(mpack_node_t node, size_t index) {
	6435	return mpack_node_map_at(node, index, 1);
	6436	}
	6437
	6438	#endif

+7151

-0

backend/lib/mpack.h less more

	0	/**
	1	* The MIT License (MIT)
	2	*
	3	* Copyright (c) 2015-2018 Nicholas Fraser
	4	*
	5	* Permission is hereby granted, free of charge, to any person obtaining a copy
	6	* of this software and associated documentation files (the "Software"), to deal
	7	* in the Software without restriction, including without limitation the rights
	8	* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	9	* copies of the Software, and to permit persons to whom the Software is
	10	* furnished to do so, subject to the following conditions:
	11	*
	12	* The above copyright notice and this permission notice shall be included in all
	13	* copies or substantial portions of the Software.
	14	*
	15	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	16	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	17	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	18	* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	19	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	20	* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
	21	* SOFTWARE.
	22	*
	23	*/
	24
	25	/*
	26	* This is the MPack 1.0 amalgamation package.
	27	*
	28	* http://github.com/ludocode/mpack
	29	*/
	30
	31	#ifndef MPACK_H
	32	#define MPACK_H 1
	33
	34	#define MPACK_AMALGAMATED 1
	35	#define MPACK_RELEASE_VERSION 1
	36
	37	#if defined(MPACK_HAS_CONFIG) && MPACK_HAS_CONFIG
	38	#include "mpack-config.h"
	39	#endif
	40
	41
	42	/* mpack/mpack-defaults.h.h */
	43
	44
	45	/**
	46	* @name Features
	47	* @{
	48	*/
	49
	50	/**
	51	* @def MPACK_READER
	52	*
	53	* Enables compilation of the base Tag Reader.
	54	*/
	55	#ifndef MPACK_READER
	56	#define MPACK_READER 1
	57	#endif
	58
	59	/**
	60	* @def MPACK_EXPECT
	61	*
	62	* Enables compilation of the static Expect API.
	63	*/
	64	#ifndef MPACK_EXPECT
	65	#define MPACK_EXPECT 1
	66	#endif
	67
	68	/**
	69	* @def MPACK_NODE
	70	*
	71	* Enables compilation of the dynamic Node API.
	72	*/
	73	#ifndef MPACK_NODE
	74	#define MPACK_NODE 1
	75	#endif
	76
	77	/**
	78	* @def MPACK_WRITER
	79	*
	80	* Enables compilation of the Writer.
	81	*/
	82	#ifndef MPACK_WRITER
	83	#define MPACK_WRITER 1
	84	#endif
	85
	86	/**
	87	* @def MPACK_COMPATIBILITY
	88	*
	89	* Enables compatibility features for reading and writing older
	90	* versions of MessagePack.
	91	*
	92	* This is disabled by default. When disabled, the behaviour is equivalent to
	93	* using the default version, @ref mpack_version_current.
	94	*
	95	* Enable this if you need to interoperate with applications or data that do
	96	* not support the new (v5) MessagePack spec. See the section on v4
	97	* compatibility in @ref docs/protocol.md for more information.
	98	*/
	99	#ifndef MPACK_COMPATIBILITY
	100	#define MPACK_COMPATIBILITY 0
	101	#endif
	102
	103	/**
	104	* @def MPACK_EXTENSIONS
	105	*
	106	* Enables the use of extension types.
	107	*
	108	* This is disabled by default. Define it to 1 to enable it. If disabled,
	109	* functions to read and write extensions will not exist, and any occurrence of
	110	* extension types in parsed messages will flag @ref mpack_error_invalid.
	111	*
	112	* MPack discourages the use of extension types. See the section on extension
	113	* types in @ref docs/protocol.md for more information.
	114	*/
	115	#ifndef MPACK_EXTENSIONS
	116	#define MPACK_EXTENSIONS 0
	117	#endif
	118
	119
	120	/**
	121	* @}
	122	*/
	123
	124
	125	/**
	126	* @name Dependencies
	127	* @{
	128	*/
	129
	130	/**
	131	* @def MPACK_HAS_CONFIG
	132	*
	133	* Enables the use of an @c mpack-config.h configuration file for MPack.
	134	* This file must be in the same folder as @c mpack.h, or it must be
	135	* available from your project's include paths.
	136	*/
	137	// This goes in your project settings.
	138
	139	/**
	140	* @def MPACK_STDLIB
	141	*
	142	* Enables the use of C stdlib. This allows the library to use malloc
	143	* for debugging and in allocation helpers.
	144	*/
	145	#ifndef MPACK_STDLIB
	146	#define MPACK_STDLIB 1
	147	#endif
	148
	149	/**
	150	* @def MPACK_STDIO
	151	*
	152	* Enables the use of C stdio. This adds helpers for easily
	153	* reading/writing C files and makes debugging easier.
	154	*/
	155	#ifndef MPACK_STDIO
	156	#define MPACK_STDIO 1
	157	#endif
	158
	159	/**
	160	* @}
	161	*/
	162
	163
	164	/**
	165	* @name System Functions
	166	* @{
	167	*/
	168
	169	/**
	170	* @def MPACK_MALLOC
	171	*
	172	* Defines the memory allocation function used by MPack. This is used by
	173	* helpers for automatically allocating data the correct size, and for
	174	* debugging functions. If this macro is undefined, the allocation helpers
	175	* will not be compiled.
	176	*
	177	* The default is @c malloc() if @ref MPACK_STDLIB is enabled.
	178	*/
	179	/**
	180	* @def MPACK_FREE
	181	*
	182	* Defines the memory free function used by MPack. This is used by helpers
	183	* for automatically allocating data the correct size. If this macro is
	184	* undefined, the allocation helpers will not be compiled.
	185	*
	186	* The default is @c free() if @ref MPACK_MALLOC has not been customized and
	187	* @ref MPACK_STDLIB is enabled.
	188	*/
	189	/**
	190	* @def MPACK_REALLOC
	191	*
	192	* Defines the realloc function used by MPack. It is used by growable
	193	* buffers to resize more efficiently.
	194	*
	195	* The default is @c realloc() if @ref MPACK_MALLOC has not been customized and
	196	* @ref MPACK_STDLIB is enabled.
	197	*
	198	* This is optional, even when @ref MPACK_MALLOC is used. If @ref MPACK_MALLOC is
	199	* set and @ref MPACK_REALLOC is not, @ref MPACK_MALLOC is used with a simple copy
	200	* to grow buffers.
	201	*/
	202	#if defined(MPACK_STDLIB) && MPACK_STDLIB && !defined(MPACK_MALLOC)
	203	#define MPACK_MALLOC malloc
	204	#define MPACK_REALLOC realloc
	205	#define MPACK_FREE free
	206	#endif
	207
	208	/**
	209	* @}
	210	*/
	211
	212
	213	/**
	214	* @name Debugging Options
	215	*/
	216
	217	/**
	218	* @def MPACK_DEBUG
	219	*
	220	* Enables debug features. You may want to wrap this around your
	221	* own debug preprocs. By default, this is enabled if @c DEBUG or @c _DEBUG
	222	* are defined. (@c NDEBUG is not used since it is allowed to have
	223	* different values in different translation units.)
	224	*/
	225	#if !defined(MPACK_DEBUG) && (defined(DEBUG) \|\| defined(_DEBUG))
	226	#define MPACK_DEBUG 1
	227	#endif
	228
	229	/**
	230	* @def MPACK_STRINGS
	231	*
	232	* Enables descriptive error and type strings.
	233	*
	234	* This can be turned off (by defining it to 0) to maximize space savings
	235	* on embedded devices. If this is disabled, string functions such as
	236	* mpack_error_to_string() and mpack_type_to_string() return an empty string.
	237	*/
	238	#ifndef MPACK_STRINGS
	239	#define MPACK_STRINGS 1
	240	#endif
	241
	242	/**
	243	* Set this to 1 to implement a custom @ref mpack_assert_fail() function.
	244	* See the documentation on @ref mpack_assert_fail() for details.
	245	*
	246	* Asserts are only used when @ref MPACK_DEBUG is enabled, and can be
	247	* triggered by bugs in MPack or bugs due to incorrect usage of MPack.
	248	*/
	249	#ifndef MPACK_CUSTOM_ASSERT
	250	#define MPACK_CUSTOM_ASSERT 0
	251	#endif
	252
	253	/**
	254	* @def MPACK_READ_TRACKING
	255	*
	256	* Enables compound type size tracking for readers. This ensures that the
	257	* correct number of elements or bytes are read from a compound type.
	258	*
	259	* This is enabled by default in debug builds (provided a @c malloc() is
	260	* available.)
	261	*/
	262	#if !defined(MPACK_READ_TRACKING) && \
	263	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	264	defined(MPACK_READER) && MPACK_READER && \
	265	defined(MPACK_MALLOC)
	266	#define MPACK_READ_TRACKING 1
	267	#endif
	268
	269	/**
	270	* @def MPACK_WRITE_TRACKING
	271	*
	272	* Enables compound type size tracking for writers. This ensures that the
	273	* correct number of elements or bytes are written in a compound type.
	274	*
	275	* Note that without write tracking enabled, it is possible for buggy code
	276	* to emit invalid MessagePack without flagging an error by writing the wrong
	277	* number of elements or bytes in a compound type. With tracking enabled,
	278	* MPack will catch such errors and break on the offending line of code.
	279	*
	280	* This is enabled by default in debug builds (provided a @c malloc() is
	281	* available.)
	282	*/
	283	#if !defined(MPACK_WRITE_TRACKING) && \
	284	defined(MPACK_DEBUG) && MPACK_DEBUG && \
	285	defined(MPACK_WRITER) && MPACK_WRITER && \
	286	defined(MPACK_MALLOC)
	287	#define MPACK_WRITE_TRACKING 1
	288	#endif
	289
	290	/**
	291	* @}
	292	*/
	293
	294
	295	/**
	296	* @name Miscellaneous Options
	297	* @{
	298	*/
	299
	300	/**
	301	* Whether to optimize for size or speed.
	302	*
	303	* Optimizing for size simplifies some parsing and encoding algorithms
	304	* at the expense of speed, and saves a few kilobytes of space in the
	305	* resulting executable.
	306	*
	307	* This automatically detects -Os with GCC/Clang. Unfortunately there
	308	* doesn't seem to be a macro defined for /Os under MSVC.
	309	*/
	310	#ifndef MPACK_OPTIMIZE_FOR_SIZE
	311	#ifdef __OPTIMIZE_SIZE__
	312	#define MPACK_OPTIMIZE_FOR_SIZE 1
	313	#else
	314	#define MPACK_OPTIMIZE_FOR_SIZE 0
	315	#endif
	316	#endif
	317
	318	/**
	319	* Stack space in bytes to use when initializing a reader or writer
	320	* with a stack-allocated buffer.
	321	*/
	322	#ifndef MPACK_STACK_SIZE
	323	#define MPACK_STACK_SIZE 4096
	324	#endif
	325
	326	/**
	327	* Buffer size to use for allocated buffers (such as for a file writer.)
	328	*
	329	* Starting with a single page and growing as needed seems to
	330	* provide the best performance with minimal memory waste.
	331	* Increasing this does not improve performance even when writing
	332	* huge messages.
	333	*/
	334	#ifndef MPACK_BUFFER_SIZE
	335	#define MPACK_BUFFER_SIZE 4096
	336	#endif
	337
	338	/**
	339	* Minimum size of an allocated node page in bytes.
	340	*
	341	* The children for a given compound element must be contiguous, so
	342	* larger pages than this may be allocated as needed. (Safety checks
	343	* exist to prevent malicious data from causing too large allocations.)
	344	*
	345	* See @ref mpack_node_data_t for the size of nodes.
	346	*
	347	* Using as many nodes fit in one memory page seems to provide the
	348	* best performance, and has very little waste when parsing small
	349	* messages.
	350	*/
	351	#ifndef MPACK_NODE_PAGE_SIZE
	352	#define MPACK_NODE_PAGE_SIZE 4096
	353	#endif
	354
	355	/**
	356	* The initial depth for the node parser. When MPACK_MALLOC is available,
	357	* the node parser has no practical depth limit, and it is not recursive
	358	* so there is no risk of overflowing the call stack.
	359	*/
	360	#ifndef MPACK_NODE_INITIAL_DEPTH
	361	#define MPACK_NODE_INITIAL_DEPTH 8
	362	#endif
	363
	364	/**
	365	* The maximum depth for the node parser if @ref MPACK_MALLOC is not available.
	366	*/
	367	#ifndef MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC
	368	#define MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC 32
	369	#endif
	370
	371	/**
	372	* @}
	373	*/
	374
	375
	376	/**
	377	* @}
	378	*/
	379
	380
	381	/* mpack/mpack-platform.h.h */
	382
	383	/**
	384	* @file
	385	*
	386	* Abstracts all platform-specific code from MPack. This contains
	387	* implementations of standard C functions when libc is not available,
	388	* as well as wrappers to library functions.
	389	*/
	390
	391	#ifndef MPACK_PLATFORM_H
	392	#define MPACK_PLATFORM_H 1
	393
	394
	395
	396	/* Pre-include checks */
	397
	398	#if defined(_MSC_VER) && _MSC_VER < 1800 && !defined(__cplusplus)
	399	#error "In Visual Studio 2012 and earlier, MPack must be compiled as C++. Enable the /Tp compiler flag."
	400	#endif
	401
	402	#if defined(WIN32) && defined(MPACK_INTERNAL) && MPACK_INTERNAL
	403	#define _CRT_SECURE_NO_WARNINGS 1
	404	#endif
	405
	406
	407
	408	/* Doxygen preprocs */
	409	#if defined(MPACK_DOXYGEN) && MPACK_DOXYGEN
	410	#define MPACK_HAS_CONFIG 0
	411	// We give these their default values of 0 here even though they are defined to
	412	// 1 in the doxyfile. Doxygen will show this as the value in the docs, even
	413	// though it ignores it when parsing the rest of the source. This is what we
	414	// want, since we want the documentation to show these defaults but still
	415	// generate documentation for the functions they add when they're on.
	416	#define MPACK_COMPATIBILITY 0
	417	#define MPACK_EXTENSIONS 0
	418	#endif
	419
	420
	421
	422	/* Include the custom config file if enabled */
	423
	424	#if defined(MPACK_HAS_CONFIG) && MPACK_HAS_CONFIG
	425	/* #include "mpack-config.h" */
	426	#endif
	427
	428	/*
	429	* Now that the optional config is included, we define the defaults
	430	* for any of the configuration options and other switches that aren't
	431	* yet defined.
	432	*/
	433	#if defined(MPACK_DOXYGEN) && MPACK_DOXYGEN
	434	/* #include "mpack-defaults-doxygen.h" */
	435	#else
	436	/* #include "mpack-defaults.h" */
	437	#endif
	438
	439	/*
	440	* All remaining configuration options that have not yet been set must
	441	* be defined here in order to support -Wundef.
	442	*/
	443	#ifndef MPACK_DEBUG
	444	#define MPACK_DEBUG 0
	445	#endif
	446	#ifndef MPACK_CUSTOM_BREAK
	447	#define MPACK_CUSTOM_BREAK 0
	448	#endif
	449	#ifndef MPACK_READ_TRACKING
	450	#define MPACK_READ_TRACKING 0
	451	#endif
	452	#ifndef MPACK_WRITE_TRACKING
	453	#define MPACK_WRITE_TRACKING 0
	454	#endif
	455	#ifndef MPACK_EMIT_INLINE_DEFS
	456	#define MPACK_EMIT_INLINE_DEFS 0
	457	#endif
	458	#ifndef MPACK_AMALGAMATED
	459	#define MPACK_AMALGAMATED 0
	460	#endif
	461	#ifndef MPACK_RELEASE_VERSION
	462	#define MPACK_RELEASE_VERSION 0
	463	#endif
	464	#ifndef MPACK_INTERNAL
	465	#define MPACK_INTERNAL 0
	466	#endif
	467	#ifndef MPACK_NO_BUILTINS
	468	#define MPACK_NO_BUILTINS 0
	469	#endif
	470
	471
	472
	473	/* System headers (based on configuration) */
	474
	475	#ifndef __STDC_LIMIT_MACROS
	476	#define __STDC_LIMIT_MACROS 1
	477	#endif
	478	#ifndef __STDC_FORMAT_MACROS
	479	#define __STDC_FORMAT_MACROS 1
	480	#endif
	481	#ifndef __STDC_CONSTANT_MACROS
	482	#define __STDC_CONSTANT_MACROS 1
	483	#endif
	484
	485	#include <stddef.h>
	486	#include <stdint.h>
	487	#include <stdbool.h>
	488	#include <inttypes.h>
	489	#include <limits.h>
	490
	491	#if MPACK_STDLIB
	492	#include <string.h>
	493	#include <stdlib.h>
	494	#endif
	495
	496	#if MPACK_STDIO
	497	#include <stdio.h>
	498	#include <errno.h>
	499	#endif
	500
	501
	502
	503	/*
	504	* Header configuration
	505	*/
	506
	507	#ifdef __cplusplus
	508	#define MPACK_EXTERN_C_START extern "C" {
	509	#define MPACK_EXTERN_C_END }
	510	#else
	511	#define MPACK_EXTERN_C_START /* nothing */
	512	#define MPACK_EXTERN_C_END /* nothing */
	513	#endif
	514
	515	/* GCC versions from 4.6 to before 5.1 warn about defining a C99
	516	* non-static inline function before declaring it (see issue #20) */
	517	#ifdef __GNUC__
	518	#if (__GNUC__ == 4 && __GNUC_MINOR__ >= 6)
	519	#ifdef __cplusplus
	520	#define MPACK_DECLARED_INLINE_WARNING_START \
	521	_Pragma ("GCC diagnostic push") \
	522	_Pragma ("GCC diagnostic ignored \"-Wmissing-declarations\"")
	523	#else
	524	#define MPACK_DECLARED_INLINE_WARNING_START \
	525	_Pragma ("GCC diagnostic push") \
	526	_Pragma ("GCC diagnostic ignored \"-Wmissing-prototypes\"")
	527	#endif
	528	#define MPACK_DECLARED_INLINE_WARNING_END \
	529	_Pragma ("GCC diagnostic pop")
	530	#endif
	531	#endif
	532	#ifndef MPACK_DECLARED_INLINE_WARNING_START
	533	#define MPACK_DECLARED_INLINE_WARNING_START /* nothing */
	534	#define MPACK_DECLARED_INLINE_WARNING_END /* nothing */
	535	#endif
	536
	537	/* GCC versions before 4.8 warn about shadowing a function with a
	538	* variable that isn't a function or function pointer (like "index") */
	539	#ifdef __GNUC__
	540	#if (__GNUC__ < 4) \|\| (__GNUC__ == 4 && __GNUC_MINOR__ < 8)
	541	#define MPACK_WSHADOW_WARNING_START \
	542	_Pragma ("GCC diagnostic push") \
	543	_Pragma ("GCC diagnostic ignored \"-Wshadow\"")
	544	#define MPACK_WSHADOW_WARNING_END \
	545	_Pragma ("GCC diagnostic pop")
	546	#endif
	547	#endif
	548	#ifndef MPACK_WSHADOW_WARNING_START
	549	#define MPACK_WSHADOW_WARNING_START /* nothing */
	550	#define MPACK_WSHADOW_WARNING_END /* nothing */
	551	#endif
	552
	553	#define MPACK_HEADER_START \
	554	MPACK_EXTERN_C_START \
	555	MPACK_WSHADOW_WARNING_START \
	556	MPACK_DECLARED_INLINE_WARNING_START
	557
	558	#define MPACK_HEADER_END \
	559	MPACK_DECLARED_INLINE_WARNING_END \
	560	MPACK_WSHADOW_WARNING_END \
	561	MPACK_EXTERN_C_END
	562
	563	MPACK_HEADER_START
	564
	565
	566
	567	/* Miscellaneous helper macros */
	568
	569	#define MPACK_UNUSED(var) ((void)(var))
	570
	571	#define MPACK_STRINGIFY_IMPL(arg) #arg
	572	#define MPACK_STRINGIFY(arg) MPACK_STRINGIFY_IMPL(arg)
	573
	574	// This is a workaround for MSVC's incorrect expansion of __VA_ARGS__. It
	575	// treats __VA_ARGS__ as a single preprocessor token when passed in the
	576	// argument list of another macro unless we use an outer wrapper to expand it
	577	// lexically first. (For safety/consistency we use this in all variadic macros
	578	// that don't ignore the variadic arguments regardless of whether __VA_ARGS__
	579	// is passed to another macro.)
	580	// https://stackoverflow.com/a/32400131
	581	#define MPACK_EXPAND(x) x
	582
	583	// Extracts the first argument of a variadic macro list, where there might only
	584	// be one argument.
	585	#define MPACK_EXTRACT_ARG0_IMPL(first, ...) first
	586	#define MPACK_EXTRACT_ARG0(...) MPACK_EXPAND(MPACK_EXTRACT_ARG0_IMPL( __VA_ARGS__ , ignored))
	587
	588	// Stringifies the first argument of a variadic macro list, where there might
	589	// only be one argument.
	590	#define MPACK_STRINGIFY_ARG0_impl(first, ...) #first
	591	#define MPACK_STRINGIFY_ARG0(...) MPACK_EXPAND(MPACK_STRINGIFY_ARG0_impl( __VA_ARGS__ , ignored))
	592
	593
	594
	595	/*
	596	* Definition of inline macros.
	597	*
	598	* MPack does not use static inline in header files; only one non-inline definition
	599	* of each function should exist in the final build. This can reduce the binary size
	600	* in cases where the compiler cannot or chooses not to inline a function.
	601	* The addresses of functions should also compare equal across translation units
	602	* regardless of whether they are declared inline.
	603	*
	604	* The above requirements mean that the declaration and definition of non-trivial
	605	* inline functions must be separated so that the definitions will only
	606	* appear when necessary. In addition, three different linkage models need
	607	* to be supported:
	608	*
	609	* - The C99 model, where a standalone function is emitted only if there is any
	610	* `extern inline` or non-`inline` declaration (including the definition itself)
	611	*
	612	* - The GNU model, where an `inline` definition emits a standalone function and an
	613	* `extern inline` definition does not, regardless of other declarations
	614	*
	615	* - The C++ model, where `inline` emits a standalone function with special
	616	* (COMDAT) linkage
	617	*
	618	* The macros below wrap up everything above. All inline functions defined in header
	619	* files have a single non-inline definition emitted in the compilation of
	620	* mpack-platform.c. All inline declarations and definitions use the same MPACK_INLINE
	621	* specification to simplify the rules on when standalone functions are emitted.
	622	* Inline functions in source files are defined MPACK_STATIC_INLINE.
	623	*
	624	* Additional reading:
	625	* http://www.greenend.org.uk/rjk/tech/inline.html
	626	*/
	627
	628	#if defined(__cplusplus)
	629	// C++ rules
	630	// The linker will need COMDAT support to link C++ object files,
	631	// so we don't need to worry about emitting definitions from C++
	632	// translation units. If mpack-platform.c (or the amalgamation)
	633	// is compiled as C, its definition will be used, otherwise a
	634	// C++ definition will be used, and no other C files will emit
	635	// a defition.
	636	#define MPACK_INLINE inline
	637
	638	#elif defined(_MSC_VER)
	639	// MSVC 2013 always uses COMDAT linkage, but it doesn't treat 'inline' as a
	640	// keyword in C99 mode. (This appears to be fixed in a later version of
	641	// MSVC but we don't bother detecting it.)
	642	#define MPACK_INLINE __inline
	643	#define MPACK_STATIC_INLINE static __inline
	644
	645	#elif defined(__GNUC__) && (defined(__GNUC_GNU_INLINE__) \|\| \
	646	!defined(__GNUC_STDC_INLINE__) && !defined(__GNUC_GNU_INLINE__))
	647	// GNU rules
	648	#if MPACK_EMIT_INLINE_DEFS
	649	#define MPACK_INLINE inline
	650	#else
	651	#define MPACK_INLINE extern inline
	652	#endif
	653
	654	#else
	655	// C99 rules
	656	#if MPACK_EMIT_INLINE_DEFS
	657	#define MPACK_INLINE extern inline
	658	#else
	659	#define MPACK_INLINE inline
	660	#endif
	661	#endif
	662
	663	#ifndef MPACK_STATIC_INLINE
	664	#define MPACK_STATIC_INLINE static inline
	665	#endif
	666
	667	#ifdef MPACK_OPTIMIZE_FOR_SPEED
	668	#error "You should define MPACK_OPTIMIZE_FOR_SIZE, not MPACK_OPTIMIZE_FOR_SPEED."
	669	#endif
	670
	671
	672
	673	/*
	674	* Prevent inlining
	675	*
	676	* When a function is only used once, it is almost always inlined
	677	* automatically. This can cause poor instruction cache usage because a
	678	* function that should rarely be called (such as buffer exhaustion handling)
	679	* will get inlined into the middle of a hot code path.
	680	*/
	681
	682	#if !MPACK_NO_BUILTINS
	683	#if defined(_MSC_VER)
	684	#define MPACK_NOINLINE __declspec(noinline)
	685	#elif defined(__GNUC__) \|\| defined(__clang__)
	686	#define MPACK_NOINLINE __attribute__((noinline))
	687	#endif
	688	#endif
	689	#ifndef MPACK_NOINLINE
	690	#define MPACK_NOINLINE /* nothing */
	691	#endif
	692
	693
	694
	695	/* Some compiler-specific keywords and builtins */
	696
	697	#if !MPACK_NO_BUILTINS
	698	#if defined(__GNUC__) \|\| defined(__clang__)
	699	#define MPACK_UNREACHABLE __builtin_unreachable()
	700	#define MPACK_NORETURN(fn) fn __attribute__((noreturn))
	701	#define MPACK_RESTRICT __restrict__
	702	#elif defined(_MSC_VER)
	703	#define MPACK_UNREACHABLE __assume(0)
	704	#define MPACK_NORETURN(fn) __declspec(noreturn) fn
	705	#define MPACK_RESTRICT __restrict
	706	#endif
	707	#endif
	708
	709	#ifndef MPACK_RESTRICT
	710	#ifdef __cplusplus
	711	#define MPACK_RESTRICT /* nothing, unavailable in C++ */
	712	#else
	713	#define MPACK_RESTRICT restrict /* required in C99 */
	714	#endif
	715	#endif
	716
	717	#ifndef MPACK_UNREACHABLE
	718	#define MPACK_UNREACHABLE ((void)0)
	719	#endif
	720	#ifndef MPACK_NORETURN
	721	#define MPACK_NORETURN(fn) fn
	722	#endif
	723
	724
	725
	726	/*
	727	* Likely/unlikely
	728	*
	729	* These should only really be used when a branch is taken (or not taken) less
	730	* than 1/1000th of the time. Buffer flush checks when writing very small
	731	* elements are a good example.
	732	*/
	733
	734	#if !MPACK_NO_BUILTINS
	735	#if defined(__GNUC__) \|\| defined(__clang__)
	736	#ifndef MPACK_LIKELY
	737	#define MPACK_LIKELY(x) __builtin_expect((x),1)
	738	#endif
	739	#ifndef MPACK_UNLIKELY
	740	#define MPACK_UNLIKELY(x) __builtin_expect((x),0)
	741	#endif
	742	#endif
	743	#endif
	744
	745	#ifndef MPACK_LIKELY
	746	#define MPACK_LIKELY(x) (x)
	747	#endif
	748	#ifndef MPACK_UNLIKELY
	749	#define MPACK_UNLIKELY(x) (x)
	750	#endif
	751
	752
	753
	754	/* Static assert */
	755
	756	#ifndef MPACK_STATIC_ASSERT
	757	#if defined(__cplusplus)
	758	#if __cplusplus >= 201103L
	759	#define MPACK_STATIC_ASSERT static_assert
	760	#endif
	761	#elif defined(__STDC_VERSION__)
	762	#if __STDC_VERSION__ >= 201112L
	763	#define MPACK_STATIC_ASSERT _Static_assert
	764	#endif
	765	#endif
	766	#endif
	767
	768	#if !MPACK_NO_BUILTINS
	769	#ifndef MPACK_STATIC_ASSERT
	770	#if defined(__has_feature)
	771	#if __has_feature(cxx_static_assert)
	772	#define MPACK_STATIC_ASSERT static_assert
	773	#elif __has_feature(c_static_assert)
	774	#define MPACK_STATIC_ASSERT _Static_assert
	775	#endif
	776	#endif
	777	#endif
	778
	779	#ifndef MPACK_STATIC_ASSERT
	780	#if defined(__GNUC__)
	781	#if __GNUC__ > 4 \|\| (__GNUC__ == 4 && __GNUC_MINOR__ >= 6)
	782	#ifndef __cplusplus
	783	#if __GNUC__ >= 5
	784	#define MPACK_IGNORE_PEDANTIC "GCC diagnostic ignored \"-Wpedantic\""
	785	#else
	786	#define MPACK_IGNORE_PEDANTIC "GCC diagnostic ignored \"-pedantic\""
	787	#endif
	788	#define MPACK_STATIC_ASSERT(expr, str) do { \
	789	_Pragma ("GCC diagnostic push") \
	790	_Pragma (MPACK_IGNORE_PEDANTIC) \
	791	_Pragma ("GCC diagnostic ignored \"-Wc++-compat\"") \
	792	_Static_assert(expr, str); \
	793	_Pragma ("GCC diagnostic pop") \
	794	} while (0)
	795	#endif
	796	#endif
	797	#endif
	798	#endif
	799
	800	#ifndef MPACK_STATIC_ASSERT
	801	#ifdef _MSC_VER
	802	#if _MSC_VER >= 1600
	803	#define MPACK_STATIC_ASSERT static_assert
	804	#endif
	805	#endif
	806	#endif
	807	#endif
	808
	809	#ifndef MPACK_STATIC_ASSERT
	810	#define MPACK_STATIC_ASSERT(expr, str) (MPACK_UNUSED(sizeof(char[1 - 2*!(expr)])))
	811	#endif
	812
	813
	814
	815	/* _Generic */
	816
	817	#ifndef MPACK_HAS_GENERIC
	818	#if defined(__clang__) && defined(__has_feature)
	819	// With Clang we can test for _Generic support directly
	820	// and ignore C/C++ version
	821	#if __has_feature(c_generic_selections)
	822	#define MPACK_HAS_GENERIC 1
	823	#else
	824	#define MPACK_HAS_GENERIC 0
	825	#endif
	826	#endif
	827	#endif
	828
	829	#ifndef MPACK_HAS_GENERIC
	830	#if defined(__STDC_VERSION__)
	831	#if __STDC_VERSION__ >= 201112L
	832	#if defined(__GNUC__) && !defined(__clang__)
	833	// GCC does not have full C11 support in GCC 4.7 and 4.8
	834	#if __GNUC__ > 4 \|\| (__GNUC__ == 4 && __GNUC_MINOR__ >= 9)
	835	#define MPACK_HAS_GENERIC 1
	836	#endif
	837	#else
	838	// We hope other compilers aren't lying about C11/_Generic support
	839	#define MPACK_HAS_GENERIC 1
	840	#endif
	841	#endif
	842	#endif
	843	#endif
	844
	845	#ifndef MPACK_HAS_GENERIC
	846	#define MPACK_HAS_GENERIC 0
	847	#endif
	848
	849
	850
	851	/*
	852	* Finite Math
	853	*
	854	* -ffinite-math-only, included in -ffast-math, breaks functions that
	855	* that check for non-finite real values such as isnan() and isinf().
	856	*
	857	* We should use this to trap errors when reading data that contains
	858	* non-finite reals. This isn't currently implemented.
	859	*/
	860
	861	#ifndef MPACK_FINITE_MATH
	862	#if defined(__FINITE_MATH_ONLY__) && __FINITE_MATH_ONLY__
	863	#define MPACK_FINITE_MATH 1
	864	#endif
	865	#endif
	866
	867	#ifndef MPACK_FINITE_MATH
	868	#define MPACK_FINITE_MATH 0
	869	#endif
	870
	871
	872
	873	/*
	874	* Endianness checks
	875	*
	876	* These define MPACK_NHSWAP*() which swap network<->host byte
	877	* order when needed.
	878	*
	879	* We leave them undefined if we can't determine the endianness
	880	* at compile-time, in which case we fall back to bit-shifts.
	881	*
	882	* See the notes in mpack-common.h.
	883	*/
	884
	885	#if defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && defined(__ORDER_BIG_ENDIAN__)
	886	#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
	887	#define MPACK_NHSWAP16(x) (x)
	888	#define MPACK_NHSWAP32(x) (x)
	889	#define MPACK_NHSWAP64(x) (x)
	890	#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
	891
	892	#if !MPACK_NO_BUILTINS
	893	#if defined(__clang__)
	894	#ifdef __has_builtin
	895	// Unlike the GCC builtins, the bswap builtins in Clang
	896	// significantly improve ARM performance.
	897	#if __has_builtin(__builtin_bswap16)
	898	#define MPACK_NHSWAP16(x) __builtin_bswap16(x)
	899	#endif
	900	#if __has_builtin(__builtin_bswap32)
	901	#define MPACK_NHSWAP32(x) __builtin_bswap32(x)
	902	#endif
	903	#if __has_builtin(__builtin_bswap64)
	904	#define MPACK_NHSWAP64(x) __builtin_bswap64(x)
	905	#endif
	906	#endif
	907
	908	#elif defined(__GNUC__)
	909
	910	// The GCC bswap builtins are apparently poorly optimized on older
	911	// versions of GCC, so we set a minimum version here just in case.
	912	// http://hardwarebug.org/2010/01/14/beware-the-builtins/
	913
	914	#if __GNUC__ > 4 \|\| (__GNUC__ == 4 && __GNUC_MINOR__ >= 5)
	915	#define MPACK_NHSWAP64(x) __builtin_bswap64(x)
	916	#endif
	917
	918	// __builtin_bswap16() was not implemented on all platforms
	919	// until GCC 4.8.0:
	920	// https://gcc.gnu.org/bugzilla/show_bug.cgi?id=52624
	921	//
	922	// The 16- and 32-bit versions in GCC significantly reduce performance
	923	// on ARM with little effect on code size so we don't use them.
	924
	925	#endif
	926	#endif
	927	#endif
	928
	929	#elif defined(_MSC_VER) && defined(_WIN32) && !MPACK_NO_BUILTINS
	930
	931	// On Windows, we assume x86 and x86_64 are always little-endian.
	932	// We make no assumptions about ARM even though all current
	933	// Windows Phone devices are little-endian in case Microsoft's
	934	// compiler is ever used with a big-endian ARM device.
	935
	936	#if defined(_M_IX86) \|\| defined(_M_X64) \|\| defined(_M_AMD64)
	937	#define MPACK_NHSWAP16(x) _byteswap_ushort(x)
	938	#define MPACK_NHSWAP32(x) _byteswap_ulong(x)
	939	#define MPACK_NHSWAP64(x) _byteswap_uint64(x)
	940	#endif
	941
	942	#endif
	943
	944	#if defined(__FLOAT_WORD_ORDER__) && defined(__BYTE_ORDER__)
	945
	946	// We check where possible that the float byte order matches the
	947	// integer byte order. This is extremely unlikely to fail, but
	948	// we check anyway just in case.
	949	//
	950	// (The static assert is placed in float/double encoders instead
	951	// of here because our static assert fallback doesn't work at
	952	// file scope)
	953
	954	#define MPACK_CHECK_FLOAT_ORDER() \
	955	MPACK_STATIC_ASSERT(__FLOAT_WORD_ORDER__ == __BYTE_ORDER__, \
	956	"float byte order does not match int byte order! float/double " \
	957	"encoding is not properly implemented on this platform.")
	958
	959	#endif
	960
	961	#ifndef MPACK_CHECK_FLOAT_ORDER
	962	#define MPACK_CHECK_FLOAT_ORDER() /* nothing */
	963	#endif
	964
	965
	966	/*
	967	* Here we define mpack_assert() and mpack_break(). They both work like a normal
	968	* assertion function in debug mode, causing a trap or abort. However, on some platforms
	969	* you can safely resume execution from mpack_break(), whereas mpack_assert() is
	970	* always fatal.
	971	*
	972	* In release mode, mpack_assert() is converted to an assurance to the compiler
	973	* that the expression cannot be false (via e.g. __assume() or __builtin_unreachable())
	974	* to improve optimization where supported. There is thus no point in "safely" handling
	975	* the case of this being false. Writing mpack_assert(0) rarely makes sense (except
	976	* possibly as a default handler in a switch) since the compiler will throw away any
	977	* code after it. If at any time an mpack_assert() is not true, the behaviour is
	978	* undefined. This also means the expression is evaluated even in release.
	979	*
	980	* mpack_break() on the other hand is compiled to nothing in release. It is
	981	* used in situations where we want to highlight a programming error as early as
	982	* possible (in the debugger), but we still handle the situation safely if it
	983	* happens in release to avoid producing incorrect results (such as in
	984	* MPACK_WRITE_TRACKING.) It does not take an expression to test because it
	985	* belongs in a safe-handling block after its failing condition has been tested.
	986	*
	987	* If stdio is available, we can add a format string describing the error, and
	988	* on some compilers we can declare it noreturn to get correct results from static
	989	* analysis tools. Note that the format string and arguments are not evaluated unless
	990	* the assertion is hit.
	991	*
	992	* Note that any arguments to mpack_assert() beyond the first are only evaluated
	993	* if the expression is false (and are never evaluated in release.)
	994	*
	995	* mpack_assert_fail() and mpack_break_hit() are defined separately
	996	* because assert is noreturn and break isn't. This distinction is very
	997	* important for static analysis tools to give correct results.
	998	*/
	999
	1000	#if MPACK_DEBUG
	1001
	1002	/**
	1003	* @addtogroup config
	1004	* @{
	1005	*/
	1006	/**
	1007	* @name Debug Functions
	1008	*/
	1009	/**
	1010	* Implement this and define @ref MPACK_CUSTOM_ASSERT to use a custom
	1011	* assertion function.
	1012	*
	1013	* This function should not return. If it does, MPack will @c abort().
	1014	*
	1015	* If you use C++, make sure you include @c mpack.h where you define
	1016	* this to get the correct linkage (or define it <code>extern "C"</code>.)
	1017	*
	1018	* Asserts are only used when @ref MPACK_DEBUG is enabled, and can be
	1019	* triggered by bugs in MPack or bugs due to incorrect usage of MPack.
	1020	*/
	1021	void mpack_assert_fail(const char* message);
	1022	/**
	1023	* @}
	1024	*/
	1025	/**
	1026	* @}
	1027	*/
	1028
	1029	MPACK_NORETURN(void mpack_assert_fail_wrapper(const char* message));
	1030	#if MPACK_STDIO
	1031	MPACK_NORETURN(void mpack_assert_fail_format(const char* format, ...));
	1032	#define mpack_assert_fail_at(line, file, exprstr, format, ...) \
	1033	MPACK_EXPAND(mpack_assert_fail_format("mpack assertion failed at " file ":" #line "\n%s\n" format, exprstr, __VA_ARGS__))
	1034	#else
	1035	#define mpack_assert_fail_at(line, file, exprstr, format, ...) \
	1036	mpack_assert_fail_wrapper("mpack assertion failed at " file ":" #line "\n" exprstr "\n")
	1037	#endif
	1038
	1039	#define mpack_assert_fail_pos(line, file, exprstr, expr, ...) \
	1040	MPACK_EXPAND(mpack_assert_fail_at(line, file, exprstr, __VA_ARGS__))
	1041
	1042	// This contains a workaround to the pedantic C99 requirement of having at
	1043	// least one argument to a variadic macro. The first argument is the
	1044	// boolean expression, the optional second argument (if provided) must be a
	1045	// literal format string, and any additional arguments are the format
	1046	// argument list.
	1047	//
	1048	// Unfortunately this means macros are expanded in the expression before it
	1049	// gets stringified. I haven't found a workaround to this.
	1050	//
	1051	// This adds two unused arguments to the format argument list when a
	1052	// format string is provided, so this would complicate the use of
	1053	// -Wformat and __attribute__((format)) on mpack_assert_fail_format() if we
	1054	// ever bothered to implement it.
	1055	#define mpack_assert(...) \
	1056	MPACK_EXPAND(((!(MPACK_EXTRACT_ARG0(__VA_ARGS__))) ? \
	1057	mpack_assert_fail_pos(__LINE__, __FILE__, MPACK_STRINGIFY_ARG0(__VA_ARGS__) , __VA_ARGS__ , "", NULL) : \
	1058	(void)0))
	1059
	1060	void mpack_break_hit(const char* message);
	1061	#if MPACK_STDIO
	1062	void mpack_break_hit_format(const char* format, ...);
	1063	#define mpack_break_hit_at(line, file, ...) \
	1064	MPACK_EXPAND(mpack_break_hit_format("mpack breakpoint hit at " file ":" #line "\n" __VA_ARGS__))
	1065	#else
	1066	#define mpack_break_hit_at(line, file, ...) \
	1067	mpack_break_hit("mpack breakpoint hit at " file ":" #line )
	1068	#endif
	1069	#define mpack_break_hit_pos(line, file, ...) MPACK_EXPAND(mpack_break_hit_at(line, file, __VA_ARGS__))
	1070	#define mpack_break(...) MPACK_EXPAND(mpack_break_hit_pos(__LINE__, __FILE__, __VA_ARGS__))
	1071	#else
	1072	#define mpack_assert(...) \
	1073	(MPACK_EXPAND((!(MPACK_EXTRACT_ARG0(__VA_ARGS__))) ? \
	1074	(MPACK_UNREACHABLE, (void)0) : \
	1075	(void)0))
	1076	#define mpack_break(...) ((void)0)
	1077	#endif
	1078
	1079
	1080
	1081	/* Wrap some needed libc functions */
	1082
	1083	#if MPACK_STDLIB
	1084	#define mpack_memcmp memcmp
	1085	#define mpack_memcpy memcpy
	1086	#define mpack_memmove memmove
	1087	#define mpack_memset memset
	1088	#ifndef mpack_strlen
	1089	#define mpack_strlen strlen
	1090	#endif
	1091
	1092	#if defined(MPACK_UNIT_TESTS) && MPACK_INTERNAL && defined(__GNUC__)
	1093	// make sure we don't use the stdlib directly during development
	1094	#undef memcmp
	1095	#undef memcpy
	1096	#undef memmove
	1097	#undef memset
	1098	#undef strlen
	1099	#undef malloc
	1100	#undef free
	1101	#pragma GCC poison memcmp
	1102	#pragma GCC poison memcpy
	1103	#pragma GCC poison memmove
	1104	#pragma GCC poison memset
	1105	#pragma GCC poison strlen
	1106	#pragma GCC poison malloc
	1107	#pragma GCC poison free
	1108	#endif
	1109
	1110	#elif defined(__GNUC__) && !MPACK_NO_BUILTINS
	1111	// there's not always a builtin memmove for GCC,
	1112	// and we don't have a way to test for it
	1113	#define mpack_memcmp __builtin_memcmp
	1114	#define mpack_memcpy __builtin_memcpy
	1115	#define mpack_memset __builtin_memset
	1116	#define mpack_strlen __builtin_strlen
	1117
	1118	#elif defined(__clang__) && defined(__has_builtin) && !MPACK_NO_BUILTINS
	1119	#if __has_builtin(__builtin_memcmp)
	1120	#define mpack_memcmp __builtin_memcmp
	1121	#endif
	1122	#if __has_builtin(__builtin_memcpy)
	1123	#define mpack_memcpy __builtin_memcpy
	1124	#endif
	1125	#if __has_builtin(__builtin_memmove)
	1126	#define mpack_memmove __builtin_memmove
	1127	#endif
	1128	#if __has_builtin(__builtin_memset)
	1129	#define mpack_memset __builtin_memset
	1130	#endif
	1131	#if __has_builtin(__builtin_strlen)
	1132	#define mpack_strlen __builtin_strlen
	1133	#endif
	1134	#endif
	1135
	1136	#ifndef mpack_memcmp
	1137	int mpack_memcmp(const void* s1, const void* s2, size_t n);
	1138	#endif
	1139	#ifndef mpack_memcpy
	1140	void* mpack_memcpy(void* MPACK_RESTRICT s1, const void* MPACK_RESTRICT s2, size_t n);
	1141	#endif
	1142	#ifndef mpack_memmove
	1143	void* mpack_memmove(void* s1, const void* s2, size_t n);
	1144	#endif
	1145	#ifndef mpack_memset
	1146	void* mpack_memset(void* s, int c, size_t n);
	1147	#endif
	1148	#ifndef mpack_strlen
	1149	size_t mpack_strlen(const char* s);
	1150	#endif
	1151
	1152	#if MPACK_STDIO
	1153	#if defined(WIN32)
	1154	#define mpack_snprintf _snprintf
	1155	#else
	1156	#define mpack_snprintf snprintf
	1157	#endif
	1158	#endif
	1159
	1160
	1161
	1162	/* Debug logging */
	1163	#if 0
	1164	#include <stdio.h>
	1165	#define mpack_log(...) (MPACK_EXPAND(printf(__VA_ARGS__), fflush(stdout)))
	1166	#else
	1167	#define mpack_log(...) ((void)0)
	1168	#endif
	1169
	1170
	1171
	1172	/* Make sure our configuration makes sense */
	1173	#if defined(MPACK_MALLOC) && !defined(MPACK_FREE)
	1174	#error "MPACK_MALLOC requires MPACK_FREE."
	1175	#endif
	1176	#if !defined(MPACK_MALLOC) && defined(MPACK_FREE)
	1177	#error "MPACK_FREE requires MPACK_MALLOC."
	1178	#endif
	1179	#if MPACK_READ_TRACKING && !defined(MPACK_READER)
	1180	#error "MPACK_READ_TRACKING requires MPACK_READER."
	1181	#endif
	1182	#if MPACK_WRITE_TRACKING && !defined(MPACK_WRITER)
	1183	#error "MPACK_WRITE_TRACKING requires MPACK_WRITER."
	1184	#endif
	1185	#ifndef MPACK_MALLOC
	1186	#if MPACK_STDIO
	1187	#error "MPACK_STDIO requires preprocessor definitions for MPACK_MALLOC and MPACK_FREE."
	1188	#endif
	1189	#if MPACK_READ_TRACKING
	1190	#error "MPACK_READ_TRACKING requires preprocessor definitions for MPACK_MALLOC and MPACK_FREE."
	1191	#endif
	1192	#if MPACK_WRITE_TRACKING
	1193	#error "MPACK_WRITE_TRACKING requires preprocessor definitions for MPACK_MALLOC and MPACK_FREE."
	1194	#endif
	1195	#endif
	1196
	1197
	1198
	1199	/* Implement realloc if unavailable */
	1200	#ifdef MPACK_MALLOC
	1201	#ifdef MPACK_REALLOC
	1202	static inline void* mpack_realloc(void* old_ptr, size_t used_size, size_t new_size) {
	1203	MPACK_UNUSED(used_size);
	1204	return MPACK_REALLOC(old_ptr, new_size);
	1205	}
	1206	#else
	1207	void* mpack_realloc(void* old_ptr, size_t used_size, size_t new_size);
	1208	#endif
	1209	#endif
	1210
	1211
	1212
	1213	/**
	1214	* @}
	1215	*/
	1216
	1217	MPACK_HEADER_END
	1218
	1219	#endif
	1220
	1221
	1222	/* mpack/mpack-common.h.h */
	1223
	1224	/**
	1225	* @file
	1226	*
	1227	* Defines types and functions shared by the MPack reader and writer.
	1228	*/
	1229
	1230	#ifndef MPACK_COMMON_H
	1231	#define MPACK_COMMON_H 1
	1232
	1233	/* #include "mpack-platform.h" */
	1234
	1235	#ifndef MPACK_PRINT_BYTE_COUNT
	1236	#define MPACK_PRINT_BYTE_COUNT 12
	1237	#endif
	1238
	1239	MPACK_HEADER_START
	1240
	1241
	1242
	1243	/**
	1244	* @defgroup common Tags and Common Elements
	1245	*
	1246	* Contains types, constants and functions shared by both the encoding
	1247	* and decoding portions of MPack.
	1248	*
	1249	* @{
	1250	*/
	1251
	1252	/* Version information */
	1253
	1254	#define MPACK_VERSION_MAJOR 1 /*< The major version number of MPack. /
	1255	#define MPACK_VERSION_MINOR 0 /*< The minor version number of MPack. /
	1256	#define MPACK_VERSION_PATCH 0 /*< The patch version number of MPack. /
	1257
	1258	/** A number containing the version number of MPack for comparison purposes. */
	1259	#define MPACK_VERSION ((MPACK_VERSION_MAJOR * 10000) + \
	1260	(MPACK_VERSION_MINOR * 100) + MPACK_VERSION_PATCH)
	1261
	1262	/** A macro to test for a minimum version of MPack. */
	1263	#define MPACK_VERSION_AT_LEAST(major, minor, patch) \
	1264	(MPACK_VERSION >= (((major) * 10000) + ((minor) * 100) + (patch)))
	1265
	1266	/** @cond */
	1267	#if (MPACK_VERSION_PATCH > 0)
	1268	#define MPACK_VERSION_STRING_BASE \
	1269	MPACK_STRINGIFY(MPACK_VERSION_MAJOR) "." \
	1270	MPACK_STRINGIFY(MPACK_VERSION_MINOR) "." \
	1271	MPACK_STRINGIFY(MPACK_VERSION_PATCH)
	1272	#else
	1273	#define MPACK_VERSION_STRING_BASE \
	1274	MPACK_STRINGIFY(MPACK_VERSION_MAJOR) "." \
	1275	MPACK_STRINGIFY(MPACK_VERSION_MINOR)
	1276	#endif
	1277	/** @endcond */
	1278
	1279	/**
	1280	* @def MPACK_VERSION_STRING
	1281	* @hideinitializer
	1282	*
	1283	* A string containing the MPack version.
	1284	*/
	1285	#if MPACK_RELEASE_VERSION
	1286	#define MPACK_VERSION_STRING MPACK_VERSION_STRING_BASE
	1287	#else
	1288	#define MPACK_VERSION_STRING MPACK_VERSION_STRING_BASE "dev"
	1289	#endif
	1290
	1291	/**
	1292	* @def MPACK_LIBRARY_STRING
	1293	* @hideinitializer
	1294	*
	1295	* A string describing MPack, containing the library name, version and debug mode.
	1296	*/
	1297	#if MPACK_DEBUG
	1298	#define MPACK_LIBRARY_STRING "MPack " MPACK_VERSION_STRING "-debug"
	1299	#else
	1300	#define MPACK_LIBRARY_STRING "MPack " MPACK_VERSION_STRING
	1301	#endif
	1302
	1303	/** @cond */
	1304	/**
	1305	* @def MPACK_MAXIMUM_TAG_SIZE
	1306	*
	1307	* The maximum encoded size of a tag in bytes.
	1308	*/
	1309	#define MPACK_MAXIMUM_TAG_SIZE 9
	1310	/** @endcond */
	1311
	1312	#if MPACK_EXTENSIONS
	1313	/**
	1314	* @def MPACK_TIMESTAMP_NANOSECONDS_MAX
	1315	*
	1316	* The maximum value of nanoseconds for a timestamp.
	1317	*
	1318	* @note This requires @ref MPACK_EXTENSIONS.
	1319	*/
	1320	#define MPACK_TIMESTAMP_NANOSECONDS_MAX 999999999
	1321	#endif
	1322
	1323
	1324
	1325	#if MPACK_COMPATIBILITY
	1326	/**
	1327	* Versions of the MessagePack format.
	1328	*
	1329	* A reader, writer, or tree can be configured to serialize in an older
	1330	* version of the MessagePack spec. This is necessary to interface with
	1331	* older MessagePack libraries that do not support new MessagePack features.
	1332	*
	1333	* @note This requires @ref MPACK_COMPATIBILITY.
	1334	*/
	1335	typedef enum mpack_version_t {
	1336
	1337	/**
	1338	* Version 1.0/v4, supporting only the @c raw type without @c str8.
	1339	*/
	1340	mpack_version_v4 = 4,
	1341
	1342	/**
	1343	* Version 2.0/v5, supporting the @c str8, @c bin and @c ext types.
	1344	*/
	1345	mpack_version_v5 = 5,
	1346
	1347	/**
	1348	* The most recent supported version of MessagePack. This is the default.
	1349	*/
	1350	mpack_version_current = mpack_version_v5,
	1351
	1352	} mpack_version_t;
	1353	#endif
	1354
	1355	/**
	1356	* Error states for MPack objects.
	1357	*
	1358	* When a reader, writer, or tree is in an error state, all subsequent calls
	1359	* are ignored and their return values are nil/zero. You should check whether
	1360	* the source is in an error state before using such values.
	1361	*/
	1362	typedef enum mpack_error_t {
	1363	mpack_ok = 0, /*< No error. /
	1364	mpack_error_io = 2, /*< The reader or writer failed to fill or flush, or some other file or socket error occurred. /
	1365	mpack_error_invalid, /*< The data read is not valid MessagePack. /
	1366	mpack_error_unsupported, /*< The data read is not supported by this configuration of MPack. (See @ref MPACK_EXTENSIONS.) /
	1367	mpack_error_type, /*< The type or value range did not match what was expected by the caller. /
	1368	mpack_error_too_big, /*< A read or write was bigger than the maximum size allowed for that operation. /
	1369	mpack_error_memory, /*< An allocation failure occurred. /
	1370	mpack_error_bug, /*< The MPack API was used incorrectly. (This will always assert in debug mode.) /
	1371	mpack_error_data, /*< The contained data is not valid. /
	1372	mpack_error_eof, /*< The reader failed to read because of file or socket EOF /
	1373	} mpack_error_t;
	1374
	1375	/**
	1376	* Converts an MPack error to a string. This function returns an empty
	1377	* string when MPACK_DEBUG is not set.
	1378	*/
	1379	const char* mpack_error_to_string(mpack_error_t error);
	1380
	1381	/**
	1382	* Defines the type of a MessagePack tag.
	1383	*
	1384	* Note that extension types, both user defined and built-in, are represented
	1385	* in tags as @ref mpack_type_ext. The value for an extension type is stored
	1386	* separately.
	1387	*/
	1388	typedef enum mpack_type_t {
	1389	mpack_type_missing = 0, /*< Special type indicating a missing optional value. /
	1390	mpack_type_nil, /*< A null value. /
	1391	mpack_type_bool, /*< A boolean (true or false.) /
	1392	mpack_type_int, /*< A 64-bit signed integer. /
	1393	mpack_type_uint, /*< A 64-bit unsigned integer. /
	1394	mpack_type_float, /*< A 32-bit IEEE 754 floating point number. /
	1395	mpack_type_double, /*< A 64-bit IEEE 754 floating point number. /
	1396	mpack_type_str, /*< A string. /
	1397	mpack_type_bin, /*< A chunk of binary data. /
	1398	mpack_type_array, /*< An array of MessagePack objects. /
	1399	mpack_type_map, /*< An ordered map of key/value pairs of MessagePack objects. /
	1400
	1401	#if MPACK_EXTENSIONS
	1402	/**
	1403	* A typed MessagePack extension object containing a chunk of binary data.
	1404	*
	1405	* @note This requires @ref MPACK_EXTENSIONS.
	1406	*/
	1407	mpack_type_ext,
	1408	#endif
	1409	} mpack_type_t;
	1410
	1411	/**
	1412	* Converts an MPack type to a string. This function returns an empty
	1413	* string when MPACK_DEBUG is not set.
	1414	*/
	1415	const char* mpack_type_to_string(mpack_type_t type);
	1416
	1417	#if MPACK_EXTENSIONS
	1418	/**
	1419	* A timestamp.
	1420	*
	1421	* @note This requires @ref MPACK_EXTENSIONS.
	1422	*/
	1423	typedef struct mpack_timestamp_t {
	1424	int64_t seconds; /< The number of seconds (signed) since 1970-01-01T00:00:00Z. /
	1425	uint32_t nanoseconds; /< The number of additional nanoseconds, between 0 and 999,999,999. /
	1426	} mpack_timestamp_t;
	1427	#endif
	1428
	1429	/**
	1430	* An MPack tag is a MessagePack object header. It is a variant type
	1431	* representing any kind of object, and includes the length of compound types
	1432	* (e.g. map, array, string) or the value of non-compound types (e.g. boolean,
	1433	* integer, float.)
	1434	*
	1435	* If the type is compound (str, bin, ext, array or map), the contained
	1436	* elements or bytes are stored separately.
	1437	*
	1438	* This structure is opaque; its fields should not be accessed outside
	1439	* of MPack.
	1440	*/
	1441	typedef struct mpack_tag_t mpack_tag_t;
	1442
	1443	/* Hide internals from documentation */
	1444	/** @cond */
	1445	struct mpack_tag_t {
	1446	mpack_type_t type; /< The type of value. /
	1447
	1448	#if MPACK_EXTENSIONS
	1449	int8_t exttype; /< The extension type if the type is @ref mpack_type_ext. /
	1450	#endif
	1451
	1452	/* The value for non-compound types. */
	1453	union {
	1454	uint64_t u; /< The value if the type is unsigned int. /
	1455	int64_t i; /< The value if the type is signed int. /
	1456	double d; /< The value if the type is double. /
	1457	float f; /< The value if the type is float. /
	1458	bool b; /< The value if the type is bool. /
	1459
	1460	/* The number of bytes if the type is str, bin or ext. */
	1461	uint32_t l;
	1462
	1463	/* The element count if the type is an array, or the number of
	1464	key/value pairs if the type is map. */
	1465	uint32_t n;
	1466	} v;
	1467	};
	1468	/** @endcond */
	1469
	1470	/**
	1471	* @name Tag Generators
	1472	* @{
	1473	*/
	1474
	1475	/**
	1476	* @def MPACK_TAG_ZERO
	1477	*
	1478	* An @ref mpack_tag_t initializer that zeroes the given tag.
	1479	*
	1480	* @warning This does not make the tag nil! The tag's type is invalid when
	1481	* initialized this way. Use @ref mpack_tag_make_nil() to generate a nil tag.
	1482	*/
	1483	#if MPACK_EXTENSIONS
	1484	#define MPACK_TAG_ZERO {(mpack_type_t)0, 0, {0}}
	1485	#else
	1486	#define MPACK_TAG_ZERO {(mpack_type_t)0, {0}}
	1487	#endif
	1488
	1489	/** Generates a nil tag. */
	1490	MPACK_INLINE mpack_tag_t mpack_tag_make_nil(void) {
	1491	mpack_tag_t ret = MPACK_TAG_ZERO;
	1492	ret.type = mpack_type_nil;
	1493	return ret;
	1494	}
	1495
	1496	/** Generates a bool tag. */
	1497	MPACK_INLINE mpack_tag_t mpack_tag_make_bool(bool value) {
	1498	mpack_tag_t ret = MPACK_TAG_ZERO;
	1499	ret.type = mpack_type_bool;
	1500	ret.v.b = value;
	1501	return ret;
	1502	}
	1503
	1504	/** Generates a bool tag with value true. */
	1505	MPACK_INLINE mpack_tag_t mpack_tag_make_true(void) {
	1506	mpack_tag_t ret = MPACK_TAG_ZERO;
	1507	ret.type = mpack_type_bool;
	1508	ret.v.b = true;
	1509	return ret;
	1510	}
	1511
	1512	/** Generates a bool tag with value false. */
	1513	MPACK_INLINE mpack_tag_t mpack_tag_make_false(void) {
	1514	mpack_tag_t ret = MPACK_TAG_ZERO;
	1515	ret.type = mpack_type_bool;
	1516	ret.v.b = false;
	1517	return ret;
	1518	}
	1519
	1520	/** Generates a signed int tag. */
	1521	MPACK_INLINE mpack_tag_t mpack_tag_make_int(int64_t value) {
	1522	mpack_tag_t ret = MPACK_TAG_ZERO;
	1523	ret.type = mpack_type_int;
	1524	ret.v.i = value;
	1525	return ret;
	1526	}
	1527
	1528	/** Generates an unsigned int tag. */
	1529	MPACK_INLINE mpack_tag_t mpack_tag_make_uint(uint64_t value) {
	1530	mpack_tag_t ret = MPACK_TAG_ZERO;
	1531	ret.type = mpack_type_uint;
	1532	ret.v.u = value;
	1533	return ret;
	1534	}
	1535
	1536	/** Generates a float tag. */
	1537	MPACK_INLINE mpack_tag_t mpack_tag_make_float(float value) {
	1538	mpack_tag_t ret = MPACK_TAG_ZERO;
	1539	ret.type = mpack_type_float;
	1540	ret.v.f = value;
	1541	return ret;
	1542	}
	1543
	1544	/** Generates a double tag. */
	1545	MPACK_INLINE mpack_tag_t mpack_tag_make_double(double value) {
	1546	mpack_tag_t ret = MPACK_TAG_ZERO;
	1547	ret.type = mpack_type_double;
	1548	ret.v.d = value;
	1549	return ret;
	1550	}
	1551
	1552	/** Generates an array tag. */
	1553	MPACK_INLINE mpack_tag_t mpack_tag_make_array(uint32_t count) {
	1554	mpack_tag_t ret = MPACK_TAG_ZERO;
	1555	ret.type = mpack_type_array;
	1556	ret.v.n = count;
	1557	return ret;
	1558	}
	1559
	1560	/** Generates a map tag. */
	1561	MPACK_INLINE mpack_tag_t mpack_tag_make_map(uint32_t count) {
	1562	mpack_tag_t ret = MPACK_TAG_ZERO;
	1563	ret.type = mpack_type_map;
	1564	ret.v.n = count;
	1565	return ret;
	1566	}
	1567
	1568	/** Generates a str tag. */
	1569	MPACK_INLINE mpack_tag_t mpack_tag_make_str(uint32_t length) {
	1570	mpack_tag_t ret = MPACK_TAG_ZERO;
	1571	ret.type = mpack_type_str;
	1572	ret.v.l = length;
	1573	return ret;
	1574	}
	1575
	1576	/** Generates a bin tag. */
	1577	MPACK_INLINE mpack_tag_t mpack_tag_make_bin(uint32_t length) {
	1578	mpack_tag_t ret = MPACK_TAG_ZERO;
	1579	ret.type = mpack_type_bin;
	1580	ret.v.l = length;
	1581	return ret;
	1582	}
	1583
	1584	#if MPACK_EXTENSIONS
	1585	/**
	1586	* Generates an ext tag.
	1587	*
	1588	* @note This requires @ref MPACK_EXTENSIONS.
	1589	*/
	1590	MPACK_INLINE mpack_tag_t mpack_tag_make_ext(int8_t exttype, uint32_t length) {
	1591	mpack_tag_t ret = MPACK_TAG_ZERO;
	1592	ret.type = mpack_type_ext;
	1593	ret.exttype = exttype;
	1594	ret.v.l = length;
	1595	return ret;
	1596	}
	1597	#endif
	1598
	1599	/**
	1600	* @}
	1601	*/
	1602
	1603	/**
	1604	* @name Tag Querying Functions
	1605	* @{
	1606	*/
	1607
	1608	/**
	1609	* Gets the type of a tag.
	1610	*/
	1611	MPACK_INLINE mpack_type_t mpack_tag_type(mpack_tag_t* tag) {
	1612	return tag->type;
	1613	}
	1614
	1615	/**
	1616	* Gets the boolean value of a bool-type tag. The tag must be of type @ref
	1617	* mpack_type_bool.
	1618	*
	1619	* This asserts that the type in the tag is @ref mpack_type_bool. (No check is
	1620	* performed if MPACK_DEBUG is not set.)
	1621	*/
	1622	MPACK_INLINE bool mpack_tag_bool_value(mpack_tag_t* tag) {
	1623	mpack_assert(tag->type == mpack_type_bool, "tag is not a bool!");
	1624	return tag->v.b;
	1625	}
	1626
	1627	/**
	1628	* Gets the signed integer value of an int-type tag.
	1629	*
	1630	* This asserts that the type in the tag is @ref mpack_type_int. (No check is
	1631	* performed if MPACK_DEBUG is not set.)
	1632	*
	1633	* @warning This does not convert between signed and unsigned tags! A positive
	1634	* integer may be stored in a tag as either @ref mpack_type_int or @ref
	1635	* mpack_type_uint. You must check the type first; this can only be used if the
	1636	* type is @ref mpack_type_int.
	1637	*
	1638	* @see mpack_type_int
	1639	*/
	1640	MPACK_INLINE int64_t mpack_tag_int_value(mpack_tag_t* tag) {
	1641	mpack_assert(tag->type == mpack_type_int, "tag is not an int!");
	1642	return tag->v.i;
	1643	}
	1644
	1645	/**
	1646	* Gets the unsigned integer value of a uint-type tag.
	1647	*
	1648	* This asserts that the type in the tag is @ref mpack_type_uint. (No check is
	1649	* performed if MPACK_DEBUG is not set.)
	1650	*
	1651	* @warning This does not convert between signed and unsigned tags! A positive
	1652	* integer may be stored in a tag as either @ref mpack_type_int or @ref
	1653	* mpack_type_uint. You must check the type first; this can only be used if the
	1654	* type is @ref mpack_type_uint.
	1655	*
	1656	* @see mpack_type_uint
	1657	*/
	1658	MPACK_INLINE uint64_t mpack_tag_uint_value(mpack_tag_t* tag) {
	1659	mpack_assert(tag->type == mpack_type_uint, "tag is not a uint!");
	1660	return tag->v.u;
	1661	}
	1662
	1663	/**
	1664	* Gets the float value of a float-type tag.
	1665	*
	1666	* This asserts that the type in the tag is @ref mpack_type_float. (No check is
	1667	* performed if MPACK_DEBUG is not set.)
	1668	*
	1669	* @warning This does not convert between float and double tags! This can only
	1670	* be used if the type is @ref mpack_type_float.
	1671	*
	1672	* @see mpack_type_float
	1673	*/
	1674	MPACK_INLINE float mpack_tag_float_value(mpack_tag_t* tag) {
	1675	mpack_assert(tag->type == mpack_type_float, "tag is not a float!");
	1676	return tag->v.f;
	1677	}
	1678
	1679	/**
	1680	* Gets the double value of a double-type tag.
	1681	*
	1682	* This asserts that the type in the tag is @ref mpack_type_double. (No check
	1683	* is performed if MPACK_DEBUG is not set.)
	1684	*
	1685	* @warning This does not convert between float and double tags! This can only
	1686	* be used if the type is @ref mpack_type_double.
	1687	*
	1688	* @see mpack_type_double
	1689	*/
	1690	MPACK_INLINE double mpack_tag_double_value(mpack_tag_t* tag) {
	1691	mpack_assert(tag->type == mpack_type_double, "tag is not a double!");
	1692	return tag->v.d;
	1693	}
	1694
	1695	/**
	1696	* Gets the number of elements in an array tag.
	1697	*
	1698	* This asserts that the type in the tag is @ref mpack_type_array. (No check is
	1699	* performed if MPACK_DEBUG is not set.)
	1700	*
	1701	* @see mpack_type_array
	1702	*/
	1703	MPACK_INLINE uint32_t mpack_tag_array_count(mpack_tag_t* tag) {
	1704	mpack_assert(tag->type == mpack_type_array, "tag is not an array!");
	1705	return tag->v.n;
	1706	}
	1707
	1708	/**
	1709	* Gets the number of key-value pairs in a map tag.
	1710	*
	1711	* This asserts that the type in the tag is @ref mpack_type_map. (No check is
	1712	* performed if MPACK_DEBUG is not set.)
	1713	*
	1714	* @see mpack_type_map
	1715	*/
	1716	MPACK_INLINE uint32_t mpack_tag_map_count(mpack_tag_t* tag) {
	1717	mpack_assert(tag->type == mpack_type_map, "tag is not a map!");
	1718	return tag->v.n;
	1719	}
	1720
	1721	/**
	1722	* Gets the length in bytes of a str-type tag.
	1723	*
	1724	* This asserts that the type in the tag is @ref mpack_type_str. (No check is
	1725	* performed if MPACK_DEBUG is not set.)
	1726	*
	1727	* @see mpack_type_str
	1728	*/
	1729	MPACK_INLINE uint32_t mpack_tag_str_length(mpack_tag_t* tag) {
	1730	mpack_assert(tag->type == mpack_type_str, "tag is not a str!");
	1731	return tag->v.l;
	1732	}
	1733
	1734	/**
	1735	* Gets the length in bytes of a bin-type tag.
	1736	*
	1737	* This asserts that the type in the tag is @ref mpack_type_bin. (No check is
	1738	* performed if MPACK_DEBUG is not set.)
	1739	*
	1740	* @see mpack_type_bin
	1741	*/
	1742	MPACK_INLINE uint32_t mpack_tag_bin_length(mpack_tag_t* tag) {
	1743	mpack_assert(tag->type == mpack_type_bin, "tag is not a bin!");
	1744	return tag->v.l;
	1745	}
	1746
	1747	#if MPACK_EXTENSIONS
	1748	/**
	1749	* Gets the length in bytes of an ext-type tag.
	1750	*
	1751	* This asserts that the type in the tag is @ref mpack_type_ext. (No check is
	1752	* performed if MPACK_DEBUG is not set.)
	1753	*
	1754	* @note This requires @ref MPACK_EXTENSIONS.
	1755	*
	1756	* @see mpack_type_ext
	1757	*/
	1758	MPACK_INLINE uint32_t mpack_tag_ext_length(mpack_tag_t* tag) {
	1759	mpack_assert(tag->type == mpack_type_ext, "tag is not an ext!");
	1760	return tag->v.l;
	1761	}
	1762
	1763	/**
	1764	* Gets the extension type (exttype) of an ext-type tag.
	1765	*
	1766	* This asserts that the type in the tag is @ref mpack_type_ext. (No check is
	1767	* performed if MPACK_DEBUG is not set.)
	1768	*
	1769	* @note This requires @ref MPACK_EXTENSIONS.
	1770	*
	1771	* @see mpack_type_ext
	1772	*/
	1773	MPACK_INLINE int8_t mpack_tag_ext_exttype(mpack_tag_t* tag) {
	1774	mpack_assert(tag->type == mpack_type_ext, "tag is not an ext!");
	1775	return tag->exttype;
	1776	}
	1777	#endif
	1778
	1779	/**
	1780	* Gets the length in bytes of a str-, bin- or ext-type tag.
	1781	*
	1782	* This asserts that the type in the tag is @ref mpack_type_str, @ref
	1783	* mpack_type_bin or @ref mpack_type_ext. (No check is performed if MPACK_DEBUG
	1784	* is not set.)
	1785	*
	1786	* @see mpack_type_str
	1787	* @see mpack_type_bin
	1788	* @see mpack_type_ext
	1789	*/
	1790	MPACK_INLINE uint32_t mpack_tag_bytes(mpack_tag_t* tag) {
	1791	#if MPACK_EXTENSIONS
	1792	mpack_assert(tag->type == mpack_type_str \|\| tag->type == mpack_type_bin
	1793	\|\| tag->type == mpack_type_ext, "tag is not a str, bin or ext!");
	1794	#else
	1795	mpack_assert(tag->type == mpack_type_str \|\| tag->type == mpack_type_bin,
	1796	"tag is not a str or bin!");
	1797	#endif
	1798	return tag->v.l;
	1799	}
	1800
	1801	/**
	1802	* @}
	1803	*/
	1804
	1805	/**
	1806	* @name Other tag functions
	1807	* @{
	1808	*/
	1809
	1810	#if MPACK_EXTENSIONS
	1811	/**
	1812	* The extension type for a timestamp.
	1813	*
	1814	* @note This requires @ref MPACK_EXTENSIONS.
	1815	*/
	1816	#define MPACK_EXTTYPE_TIMESTAMP ((int8_t)(-1))
	1817	#endif
	1818
	1819	/**
	1820	* Compares two tags with an arbitrary fixed ordering. Returns 0 if the tags are
	1821	* equal, a negative integer if left comes before right, or a positive integer
	1822	* otherwise.
	1823	*
	1824	* \warning The ordering is not guaranteed to be preserved across MPack versions; do
	1825	* not rely on it in persistent data.
	1826	*
	1827	* \warning Floating point numbers are compared bit-for-bit, not using the language's
	1828	* operator==. This means that NaNs with matching representation will compare equal.
	1829	* This behaviour is up for debate; see comments in the definition of mpack_tag_cmp().
	1830	*
	1831	* See mpack_tag_equal() for more information on when tags are considered equal.
	1832	*/
	1833	int mpack_tag_cmp(mpack_tag_t left, mpack_tag_t right);
	1834
	1835	/**
	1836	* Compares two tags for equality. Tags are considered equal if the types are compatible
	1837	* and the values (for non-compound types) are equal.
	1838	*
	1839	* The field width of variable-width fields is ignored (and in fact is not stored
	1840	* in a tag), and positive numbers in signed integers are considered equal to their
	1841	* unsigned counterparts. So for example the value 1 stored as a positive fixint
	1842	* is equal to the value 1 stored in a 64-bit unsigned integer field.
	1843	*
	1844	* The "extension type" of an extension object is considered part of the value
	1845	* and must match exactly.
	1846	*
	1847	* \warning Floating point numbers are compared bit-for-bit, not using the language's
	1848	* operator==. This means that NaNs with matching representation will compare equal.
	1849	* This behaviour is up for debate; see comments in the definition of mpack_tag_cmp().
	1850	*/
	1851	MPACK_INLINE bool mpack_tag_equal(mpack_tag_t left, mpack_tag_t right) {
	1852	return mpack_tag_cmp(left, right) == 0;
	1853	}
	1854
	1855	#if MPACK_DEBUG && MPACK_STDIO
	1856	/**
	1857	* Generates a json-like debug description of the given tag into the given buffer.
	1858	*
	1859	* This is only available in debug mode, and only if stdio is available (since
	1860	* it uses snprintf().) It's strictly for debugging purposes.
	1861	*
	1862	* The prefix is used to print the first few hexadecimal bytes of a bin or ext
	1863	* type. Pass NULL if not a bin or ext.
	1864	*/
	1865	void mpack_tag_debug_pseudo_json(mpack_tag_t tag, char* buffer, size_t buffer_size,
	1866	const char* prefix, size_t prefix_size);
	1867
	1868	/**
	1869	* Generates a debug string description of the given tag into the given buffer.
	1870	*
	1871	* This is only available in debug mode, and only if stdio is available (since
	1872	* it uses snprintf().) It's strictly for debugging purposes.
	1873	*/
	1874	void mpack_tag_debug_describe(mpack_tag_t tag, char* buffer, size_t buffer_size);
	1875
	1876	/** @cond */
	1877
	1878	/*
	1879	* A callback function for printing pseudo-JSON for debugging purposes.
	1880	*
	1881	* @see mpack_node_print_callback
	1882	*/
	1883	typedef void (mpack_print_callback_t)(void context, const char* data, size_t count);
	1884
	1885	// helpers for printing debug output
	1886	// i feel a bit like i'm re-implementing a buffered writer again...
	1887	typedef struct mpack_print_t {
	1888	char* buffer;
	1889	size_t size;
	1890	size_t count;
	1891	mpack_print_callback_t callback;
	1892	void* context;
	1893	} mpack_print_t;
	1894
	1895	void mpack_print_append(mpack_print_t* print, const char* data, size_t count);
	1896
	1897	MPACK_INLINE void mpack_print_append_cstr(mpack_print_t* print, const char* cstr) {
	1898	mpack_print_append(print, cstr, mpack_strlen(cstr));
	1899	}
	1900
	1901	void mpack_print_flush(mpack_print_t* print);
	1902
	1903	void mpack_print_file_callback(void* context, const char* data, size_t count);
	1904
	1905	/** @endcond */
	1906
	1907	#endif
	1908
	1909	/**
	1910	* @}
	1911	*/
	1912
	1913	/**
	1914	* @name Deprecated Tag Generators
	1915	* @{
	1916	*/
	1917
	1918	/*
	1919	* "make" has been added to their names to disambiguate them from the
	1920	* value-fetching functions (e.g. mpack_tag_make_bool() vs
	1921	* mpack_tag_bool_value().)
	1922	*
	1923	* The length and count for all compound types was the wrong sign (int32_t
	1924	* instead of uint32_t.) These preserve the old behaviour; the new "make"
	1925	* functions have the correct sign.
	1926	*/
	1927
	1928	/** \deprecated Renamed to mpack_tag_make_nil(). */
	1929	MPACK_INLINE mpack_tag_t mpack_tag_nil(void) {
	1930	return mpack_tag_make_nil();
	1931	}
	1932
	1933	/** \deprecated Renamed to mpack_tag_make_bool(). */
	1934	MPACK_INLINE mpack_tag_t mpack_tag_bool(bool value) {
	1935	return mpack_tag_make_bool(value);
	1936	}
	1937
	1938	/** \deprecated Renamed to mpack_tag_make_true(). */
	1939	MPACK_INLINE mpack_tag_t mpack_tag_true(void) {
	1940	return mpack_tag_make_true();
	1941	}
	1942
	1943	/** \deprecated Renamed to mpack_tag_make_false(). */
	1944	MPACK_INLINE mpack_tag_t mpack_tag_false(void) {
	1945	return mpack_tag_make_false();
	1946	}
	1947
	1948	/** \deprecated Renamed to mpack_tag_make_int(). */
	1949	MPACK_INLINE mpack_tag_t mpack_tag_int(int64_t value) {
	1950	return mpack_tag_make_int(value);
	1951	}
	1952
	1953	/** \deprecated Renamed to mpack_tag_make_uint(). */
	1954	MPACK_INLINE mpack_tag_t mpack_tag_uint(uint64_t value) {
	1955	return mpack_tag_make_uint(value);
	1956	}
	1957
	1958	/** \deprecated Renamed to mpack_tag_make_float(). */
	1959	MPACK_INLINE mpack_tag_t mpack_tag_float(float value) {
	1960	return mpack_tag_make_float(value);
	1961	}
	1962
	1963	/** \deprecated Renamed to mpack_tag_make_double(). */
	1964	MPACK_INLINE mpack_tag_t mpack_tag_double(double value) {
	1965	return mpack_tag_make_double(value);
	1966	}
	1967
	1968	/** \deprecated Renamed to mpack_tag_make_array(). */
	1969	MPACK_INLINE mpack_tag_t mpack_tag_array(int32_t count) {
	1970	return mpack_tag_make_array((uint32_t)count);
	1971	}
	1972
	1973	/** \deprecated Renamed to mpack_tag_make_map(). */
	1974	MPACK_INLINE mpack_tag_t mpack_tag_map(int32_t count) {
	1975	return mpack_tag_make_map((uint32_t)count);
	1976	}
	1977
	1978	/** \deprecated Renamed to mpack_tag_make_str(). */
	1979	MPACK_INLINE mpack_tag_t mpack_tag_str(int32_t length) {
	1980	return mpack_tag_make_str((uint32_t)length);
	1981	}
	1982
	1983	/** \deprecated Renamed to mpack_tag_make_bin(). */
	1984	MPACK_INLINE mpack_tag_t mpack_tag_bin(int32_t length) {
	1985	return mpack_tag_make_bin((uint32_t)length);
	1986	}
	1987
	1988	#if MPACK_EXTENSIONS
	1989	/** \deprecated Renamed to mpack_tag_make_ext(). */
	1990	MPACK_INLINE mpack_tag_t mpack_tag_ext(int8_t exttype, int32_t length) {
	1991	return mpack_tag_make_ext(exttype, (uint32_t)length);
	1992	}
	1993	#endif
	1994
	1995	/**
	1996	* @}
	1997	*/
	1998
	1999	/** @cond */
	2000
	2001	/*
	2002	* Helpers to perform unaligned network-endian loads and stores
	2003	* at arbitrary addresses. Byte-swapping builtins are used if they
	2004	* are available and if they improve performance.
	2005	*
	2006	* These will remain available in the public API so feel free to
	2007	* use them for other purposes, but they are undocumented.
	2008	*/
	2009
	2010	MPACK_INLINE uint8_t mpack_load_u8(const char* p) {
	2011	return (uint8_t)p[0];
	2012	}
	2013
	2014	MPACK_INLINE uint16_t mpack_load_u16(const char* p) {
	2015	#ifdef MPACK_NHSWAP16
	2016	uint16_t val;
	2017	mpack_memcpy(&val, p, sizeof(val));
	2018	return MPACK_NHSWAP16(val);
	2019	#else
	2020	return (uint16_t)((((uint16_t)(uint8_t)p[0]) << 8) \|
	2021	((uint16_t)(uint8_t)p[1]));
	2022	#endif
	2023	}
	2024
	2025	MPACK_INLINE uint32_t mpack_load_u32(const char* p) {
	2026	#ifdef MPACK_NHSWAP32
	2027	uint32_t val;
	2028	mpack_memcpy(&val, p, sizeof(val));
	2029	return MPACK_NHSWAP32(val);
	2030	#else
	2031	return (((uint32_t)(uint8_t)p[0]) << 24) \|
	2032	(((uint32_t)(uint8_t)p[1]) << 16) \|
	2033	(((uint32_t)(uint8_t)p[2]) << 8) \|
	2034	((uint32_t)(uint8_t)p[3]);
	2035	#endif
	2036	}
	2037
	2038	MPACK_INLINE uint64_t mpack_load_u64(const char* p) {
	2039	#ifdef MPACK_NHSWAP64
	2040	uint64_t val;
	2041	mpack_memcpy(&val, p, sizeof(val));
	2042	return MPACK_NHSWAP64(val);
	2043	#else
	2044	return (((uint64_t)(uint8_t)p[0]) << 56) \|
	2045	(((uint64_t)(uint8_t)p[1]) << 48) \|
	2046	(((uint64_t)(uint8_t)p[2]) << 40) \|
	2047	(((uint64_t)(uint8_t)p[3]) << 32) \|
	2048	(((uint64_t)(uint8_t)p[4]) << 24) \|
	2049	(((uint64_t)(uint8_t)p[5]) << 16) \|
	2050	(((uint64_t)(uint8_t)p[6]) << 8) \|
	2051	((uint64_t)(uint8_t)p[7]);
	2052	#endif
	2053	}
	2054
	2055	MPACK_INLINE void mpack_store_u8(char* p, uint8_t val) {
	2056	uint8_t* u = (uint8_t*)p;
	2057	u[0] = val;
	2058	}
	2059
	2060	MPACK_INLINE void mpack_store_u16(char* p, uint16_t val) {
	2061	#ifdef MPACK_NHSWAP16
	2062	val = MPACK_NHSWAP16(val);
	2063	mpack_memcpy(p, &val, sizeof(val));
	2064	#else
	2065	uint8_t* u = (uint8_t*)p;
	2066	u[0] = (uint8_t)((val >> 8) & 0xFF);
	2067	u[1] = (uint8_t)( val & 0xFF);
	2068	#endif
	2069	}
	2070
	2071	MPACK_INLINE void mpack_store_u32(char* p, uint32_t val) {
	2072	#ifdef MPACK_NHSWAP32
	2073	val = MPACK_NHSWAP32(val);
	2074	mpack_memcpy(p, &val, sizeof(val));
	2075	#else
	2076	uint8_t* u = (uint8_t*)p;
	2077	u[0] = (uint8_t)((val >> 24) & 0xFF);
	2078	u[1] = (uint8_t)((val >> 16) & 0xFF);
	2079	u[2] = (uint8_t)((val >> 8) & 0xFF);
	2080	u[3] = (uint8_t)( val & 0xFF);
	2081	#endif
	2082	}
	2083
	2084	MPACK_INLINE void mpack_store_u64(char* p, uint64_t val) {
	2085	#ifdef MPACK_NHSWAP64
	2086	val = MPACK_NHSWAP64(val);
	2087	mpack_memcpy(p, &val, sizeof(val));
	2088	#else
	2089	uint8_t* u = (uint8_t*)p;
	2090	u[0] = (uint8_t)((val >> 56) & 0xFF);
	2091	u[1] = (uint8_t)((val >> 48) & 0xFF);
	2092	u[2] = (uint8_t)((val >> 40) & 0xFF);
	2093	u[3] = (uint8_t)((val >> 32) & 0xFF);
	2094	u[4] = (uint8_t)((val >> 24) & 0xFF);
	2095	u[5] = (uint8_t)((val >> 16) & 0xFF);
	2096	u[6] = (uint8_t)((val >> 8) & 0xFF);
	2097	u[7] = (uint8_t)( val & 0xFF);
	2098	#endif
	2099	}
	2100
	2101	MPACK_INLINE int8_t mpack_load_i8 (const char* p) {return (int8_t) mpack_load_u8 (p);}
	2102	MPACK_INLINE int16_t mpack_load_i16(const char* p) {return (int16_t)mpack_load_u16(p);}
	2103	MPACK_INLINE int32_t mpack_load_i32(const char* p) {return (int32_t)mpack_load_u32(p);}
	2104	MPACK_INLINE int64_t mpack_load_i64(const char* p) {return (int64_t)mpack_load_u64(p);}
	2105	MPACK_INLINE void mpack_store_i8 (char* p, int8_t val) {mpack_store_u8 (p, (uint8_t) val);}
	2106	MPACK_INLINE void mpack_store_i16(char* p, int16_t val) {mpack_store_u16(p, (uint16_t)val);}
	2107	MPACK_INLINE void mpack_store_i32(char* p, int32_t val) {mpack_store_u32(p, (uint32_t)val);}
	2108	MPACK_INLINE void mpack_store_i64(char* p, int64_t val) {mpack_store_u64(p, (uint64_t)val);}
	2109
	2110	MPACK_INLINE float mpack_load_float(const char* p) {
	2111	MPACK_CHECK_FLOAT_ORDER();
	2112	MPACK_STATIC_ASSERT(sizeof(float) == sizeof(uint32_t), "float is wrong size??");
	2113	union {
	2114	float f;
	2115	uint32_t u;
	2116	} v;
	2117	v.u = mpack_load_u32(p);
	2118	return v.f;
	2119	}
	2120
	2121	MPACK_INLINE double mpack_load_double(const char* p) {
	2122	MPACK_CHECK_FLOAT_ORDER();
	2123	MPACK_STATIC_ASSERT(sizeof(double) == sizeof(uint64_t), "double is wrong size??");
	2124	union {
	2125	double d;
	2126	uint64_t u;
	2127	} v;
	2128	v.u = mpack_load_u64(p);
	2129	return v.d;
	2130	}
	2131
	2132	MPACK_INLINE void mpack_store_float(char* p, float value) {
	2133	MPACK_CHECK_FLOAT_ORDER();
	2134	union {
	2135	float f;
	2136	uint32_t u;
	2137	} v;
	2138	v.f = value;
	2139	mpack_store_u32(p, v.u);
	2140	}
	2141
	2142	MPACK_INLINE void mpack_store_double(char* p, double value) {
	2143	MPACK_CHECK_FLOAT_ORDER();
	2144	union {
	2145	double d;
	2146	uint64_t u;
	2147	} v;
	2148	v.d = value;
	2149	mpack_store_u64(p, v.u);
	2150	}
	2151
	2152	/** @endcond */
	2153
	2154
	2155
	2156	/** @cond */
	2157
	2158	// Sizes in bytes for the various possible tags
	2159	#define MPACK_TAG_SIZE_FIXUINT 1
	2160	#define MPACK_TAG_SIZE_U8 2
	2161	#define MPACK_TAG_SIZE_U16 3
	2162	#define MPACK_TAG_SIZE_U32 5
	2163	#define MPACK_TAG_SIZE_U64 9
	2164	#define MPACK_TAG_SIZE_FIXINT 1
	2165	#define MPACK_TAG_SIZE_I8 2
	2166	#define MPACK_TAG_SIZE_I16 3
	2167	#define MPACK_TAG_SIZE_I32 5
	2168	#define MPACK_TAG_SIZE_I64 9
	2169	#define MPACK_TAG_SIZE_FLOAT 5
	2170	#define MPACK_TAG_SIZE_DOUBLE 9
	2171	#define MPACK_TAG_SIZE_FIXARRAY 1
	2172	#define MPACK_TAG_SIZE_ARRAY16 3
	2173	#define MPACK_TAG_SIZE_ARRAY32 5
	2174	#define MPACK_TAG_SIZE_FIXMAP 1
	2175	#define MPACK_TAG_SIZE_MAP16 3
	2176	#define MPACK_TAG_SIZE_MAP32 5
	2177	#define MPACK_TAG_SIZE_FIXSTR 1
	2178	#define MPACK_TAG_SIZE_STR8 2
	2179	#define MPACK_TAG_SIZE_STR16 3
	2180	#define MPACK_TAG_SIZE_STR32 5
	2181	#define MPACK_TAG_SIZE_BIN8 2
	2182	#define MPACK_TAG_SIZE_BIN16 3
	2183	#define MPACK_TAG_SIZE_BIN32 5
	2184	#define MPACK_TAG_SIZE_FIXEXT1 2
	2185	#define MPACK_TAG_SIZE_FIXEXT2 2
	2186	#define MPACK_TAG_SIZE_FIXEXT4 2
	2187	#define MPACK_TAG_SIZE_FIXEXT8 2
	2188	#define MPACK_TAG_SIZE_FIXEXT16 2
	2189	#define MPACK_TAG_SIZE_EXT8 3
	2190	#define MPACK_TAG_SIZE_EXT16 4
	2191	#define MPACK_TAG_SIZE_EXT32 6
	2192
	2193	// size in bytes for complete ext types
	2194	#define MPACK_EXT_SIZE_TIMESTAMP4 (MPACK_TAG_SIZE_FIXEXT4 + 4)
	2195	#define MPACK_EXT_SIZE_TIMESTAMP8 (MPACK_TAG_SIZE_FIXEXT8 + 8)
	2196	#define MPACK_EXT_SIZE_TIMESTAMP12 (MPACK_TAG_SIZE_EXT8 + 12)
	2197
	2198	/** @endcond */
	2199
	2200
	2201
	2202	#if MPACK_READ_TRACKING \|\| MPACK_WRITE_TRACKING
	2203	/* Tracks the write state of compound elements (maps, arrays, */
	2204	/* strings, binary blobs and extension types) */
	2205	/** @cond */
	2206
	2207	typedef struct mpack_track_element_t {
	2208	mpack_type_t type;
	2209	uint64_t left; // we need 64-bit because (2 * INT32_MAX) elements can be stored in a map
	2210	} mpack_track_element_t;
	2211
	2212	typedef struct mpack_track_t {
	2213	size_t count;
	2214	size_t capacity;
	2215	mpack_track_element_t* elements;
	2216	} mpack_track_t;
	2217
	2218	#if MPACK_INTERNAL
	2219	mpack_error_t mpack_track_init(mpack_track_t* track);
	2220	mpack_error_t mpack_track_grow(mpack_track_t* track);
	2221	mpack_error_t mpack_track_push(mpack_track_t* track, mpack_type_t type, uint64_t count);
	2222	mpack_error_t mpack_track_pop(mpack_track_t* track, mpack_type_t type);
	2223	mpack_error_t mpack_track_element(mpack_track_t* track, bool read);
	2224	mpack_error_t mpack_track_peek_element(mpack_track_t* track, bool read);
	2225	mpack_error_t mpack_track_bytes(mpack_track_t* track, bool read, uint64_t count);
	2226	mpack_error_t mpack_track_str_bytes_all(mpack_track_t* track, bool read, uint64_t count);
	2227	mpack_error_t mpack_track_check_empty(mpack_track_t* track);
	2228	mpack_error_t mpack_track_destroy(mpack_track_t* track, bool cancel);
	2229	#endif
	2230
	2231	/** @endcond */
	2232	#endif
	2233
	2234
	2235
	2236	#if MPACK_INTERNAL
	2237	/** @cond */
	2238
	2239
	2240
	2241	/* Miscellaneous string functions */
	2242
	2243	/**
	2244	* Returns true if the given UTF-8 string is valid.
	2245	*/
	2246	bool mpack_utf8_check(const char* str, size_t bytes);
	2247
	2248	/**
	2249	* Returns true if the given UTF-8 string is valid and contains no null characters.
	2250	*/
	2251	bool mpack_utf8_check_no_null(const char* str, size_t bytes);
	2252
	2253	/**
	2254	* Returns true if the given string has no null bytes.
	2255	*/
	2256	bool mpack_str_check_no_null(const char* str, size_t bytes);
	2257
	2258
	2259
	2260	/** @endcond */
	2261	#endif
	2262
	2263
	2264
	2265	/**
	2266	* @}
	2267	*/
	2268
	2269	/**
	2270	* @}
	2271	*/
	2272
	2273	MPACK_HEADER_END
	2274
	2275	#endif
	2276
	2277
	2278	/* mpack/mpack-writer.h.h */
	2279
	2280	/**
	2281	* @file
	2282	*
	2283	* Declares the MPack Writer.
	2284	*/
	2285
	2286	#ifndef MPACK_WRITER_H
	2287	#define MPACK_WRITER_H 1
	2288
	2289	/* #include "mpack-common.h" */
	2290
	2291	MPACK_HEADER_START
	2292
	2293	#if MPACK_WRITER
	2294
	2295	#if MPACK_WRITE_TRACKING
	2296	struct mpack_track_t;
	2297	#endif
	2298
	2299	/**
	2300	* @defgroup writer Write API
	2301	*
	2302	* The MPack Write API encodes structured data of a fixed (hardcoded) schema to MessagePack.
	2303	*
	2304	* @{
	2305	*/
	2306
	2307	/**
	2308	* @def MPACK_WRITER_MINIMUM_BUFFER_SIZE
	2309	*
	2310	* The minimum buffer size for a writer with a flush function.
	2311	*/
	2312	#define MPACK_WRITER_MINIMUM_BUFFER_SIZE 32
	2313
	2314	/**
	2315	* A buffered MessagePack encoder.
	2316	*
	2317	* The encoder wraps an existing buffer and, optionally, a flush function.
	2318	* This allows efficiently encoding to an in-memory buffer or to a stream.
	2319	*
	2320	* All write operations are synchronous; they will block until the
	2321	* data is fully written, or an error occurs.
	2322	*/
	2323	typedef struct mpack_writer_t mpack_writer_t;
	2324
	2325	/**
	2326	* The MPack writer's flush function to flush the buffer to the output stream.
	2327	* It should flag an appropriate error on the writer if flushing fails (usually
	2328	* mpack_error_io or mpack_error_memory.)
	2329	*
	2330	* The specified context for callbacks is at writer->context.
	2331	*/
	2332	typedef void (mpack_writer_flush_t)(mpack_writer_t writer, const char* buffer, size_t count);
	2333
	2334	/**
	2335	* An error handler function to be called when an error is flagged on
	2336	* the writer.
	2337	*
	2338	* The error handler will only be called once on the first error flagged;
	2339	* any subsequent writes and errors are ignored, and the writer is
	2340	* permanently in that error state.
	2341	*
	2342	* MPack is safe against non-local jumps out of error handler callbacks.
	2343	* This means you are allowed to longjmp or throw an exception (in C++,
	2344	* Objective-C, or with SEH) out of this callback.
	2345	*
	2346	* Bear in mind when using longjmp that local non-volatile variables that
	2347	* have changed are undefined when setjmp() returns, so you can't put the
	2348	* writer on the stack in the same activation frame as the setjmp without
	2349	* declaring it volatile.
	2350	*
	2351	* You must still eventually destroy the writer. It is not destroyed
	2352	* automatically when an error is flagged. It is safe to destroy the
	2353	* writer within this error callback, but you will either need to perform
	2354	* a non-local jump, or store something in your context to identify
	2355	* that the writer is destroyed since any future accesses to it cause
	2356	* undefined behavior.
	2357	*/
	2358	typedef void (mpack_writer_error_t)(mpack_writer_t writer, mpack_error_t error);
	2359
	2360	/**
	2361	* A teardown function to be called when the writer is destroyed.
	2362	*/
	2363	typedef void (mpack_writer_teardown_t)(mpack_writer_t writer);
	2364
	2365	/* Hide internals from documentation */
	2366	/** @cond */
	2367
	2368	struct mpack_writer_t {
	2369	#if MPACK_COMPATIBILITY
	2370	mpack_version_t version; /* Version of the MessagePack spec to write */
	2371	#endif
	2372	mpack_writer_flush_t flush; /* Function to write bytes to the output stream */
	2373	mpack_writer_error_t error_fn; /* Function to call on error */
	2374	mpack_writer_teardown_t teardown; /* Function to teardown the context on destroy */
	2375	void* context; /* Context for writer callbacks */
	2376
	2377	char* buffer; /* Byte buffer */
	2378	char* current; /* Current position within the buffer */
	2379	char* end; /* The end of the buffer */
	2380	mpack_error_t error; /* Error state */
	2381
	2382	#if MPACK_WRITE_TRACKING
	2383	mpack_track_t track; /* Stack of map/array/str/bin/ext writes */
	2384	#endif
	2385
	2386	#ifdef MPACK_MALLOC
	2387	/* Reserved. You can use this space to allocate a custom
	2388	* context in order to reduce heap allocations. */
	2389	void* reserved[2];
	2390	#endif
	2391	};
	2392
	2393	#if MPACK_WRITE_TRACKING
	2394	void mpack_writer_track_push(mpack_writer_t* writer, mpack_type_t type, uint64_t count);
	2395	void mpack_writer_track_pop(mpack_writer_t* writer, mpack_type_t type);
	2396	void mpack_writer_track_element(mpack_writer_t* writer);
	2397	void mpack_writer_track_bytes(mpack_writer_t* writer, size_t count);
	2398	#else
	2399	MPACK_INLINE void mpack_writer_track_push(mpack_writer_t* writer, mpack_type_t type, uint64_t count) {
	2400	MPACK_UNUSED(writer);
	2401	MPACK_UNUSED(type);
	2402	MPACK_UNUSED(count);
	2403	}
	2404	MPACK_INLINE void mpack_writer_track_pop(mpack_writer_t* writer, mpack_type_t type) {
	2405	MPACK_UNUSED(writer);
	2406	MPACK_UNUSED(type);
	2407	}
	2408	MPACK_INLINE void mpack_writer_track_element(mpack_writer_t* writer) {
	2409	MPACK_UNUSED(writer);
	2410	}
	2411	MPACK_INLINE void mpack_writer_track_bytes(mpack_writer_t* writer, size_t count) {
	2412	MPACK_UNUSED(writer);
	2413	MPACK_UNUSED(count);
	2414	}
	2415	#endif
	2416
	2417	/** @endcond */
	2418
	2419	/**
	2420	* @name Lifecycle Functions
	2421	* @{
	2422	*/
	2423
	2424	/**
	2425	* Initializes an MPack writer with the given buffer. The writer
	2426	* does not assume ownership of the buffer.
	2427	*
	2428	* Trying to write past the end of the buffer will result in mpack_error_too_big
	2429	* unless a flush function is set with mpack_writer_set_flush(). To use the data
	2430	* without flushing, call mpack_writer_buffer_used() to determine the number of
	2431	* bytes written.
	2432	*
	2433	* @param writer The MPack writer.
	2434	* @param buffer The buffer into which to write MessagePack data.
	2435	* @param size The size of the buffer.
	2436	*/
	2437	void mpack_writer_init(mpack_writer_t* writer, char* buffer, size_t size);
	2438
	2439	#ifdef MPACK_MALLOC
	2440	/**
	2441	* Initializes an MPack writer using a growable buffer.
	2442	*
	2443	* The data is placed in the given data pointer if and when the writer
	2444	* is destroyed without error. The data pointer is NULL during writing,
	2445	* and will remain NULL if an error occurs.
	2446	*
	2447	* The allocated data must be freed with MPACK_FREE() (or simply free()
	2448	* if MPack's allocator hasn't been customized.)
	2449	*
	2450	* @throws mpack_error_memory if the buffer fails to grow when
	2451	* flushing.
	2452	*
	2453	* @param writer The MPack writer.
	2454	* @param data Where to place the allocated data.
	2455	* @param size Where to write the size of the data.
	2456	*/
	2457	void mpack_writer_init_growable(mpack_writer_t* writer, char** data, size_t* size);
	2458	#endif
	2459
	2460	/**
	2461	* Initializes an MPack writer directly into an error state. Use this if you
	2462	* are writing a wrapper to mpack_writer_init() which can fail its setup.
	2463	*/
	2464	void mpack_writer_init_error(mpack_writer_t* writer, mpack_error_t error);
	2465
	2466	#if MPACK_STDIO
	2467	/**
	2468	* Initializes an MPack writer that writes to a file.
	2469	*
	2470	* @throws mpack_error_memory if allocation fails
	2471	* @throws mpack_error_io if the file cannot be opened
	2472	*/
	2473	void mpack_writer_init_filename(mpack_writer_t* writer, const char* filename);
	2474
	2475	/**
	2476	* Deprecated.
	2477	*
	2478	* \deprecated Renamed to mpack_writer_init_filename().
	2479	*/
	2480	MPACK_INLINE void mpack_writer_init_file(mpack_writer_t* writer, const char* filename) {
	2481	mpack_writer_init_filename(writer, filename);
	2482	}
	2483
	2484	/**
	2485	* Initializes an MPack writer that writes to a libc FILE. This can be used to
	2486	* write to stdout or stderr, or to a file opened separately.
	2487	*
	2488	* @param writer The MPack writer.
	2489	* @param stdfile The FILE.
	2490	* @param close_when_done If true, fclose() will be called on the FILE when it
	2491	* is no longer needed. If false, the file will not be flushed or
	2492	* closed when writing is done.
	2493	*
	2494	* @note The writer is buffered. If you want to write other data to the FILE in
	2495	* between messages, you must flush it first.
	2496	*
	2497	* @see mpack_writer_flush_message
	2498	*/
	2499	void mpack_writer_init_stdfile(mpack_writer_t* writer, FILE* stdfile, bool close_when_done);
	2500	#endif
	2501
	2502	/** @cond */
	2503
	2504	#define mpack_writer_init_stack_line_ex(line, writer) \
	2505	char mpack_buf_##line[MPACK_STACK_SIZE]; \
	2506	mpack_writer_init(writer, mpack_buf_##line, sizeof(mpack_buf_##line))
	2507
	2508	#define mpack_writer_init_stack_line(line, writer) \
	2509	mpack_writer_init_stack_line_ex(line, writer)
	2510
	2511	/*
	2512	* Initializes an MPack writer using stack space as a buffer. A flush function
	2513	* should be added to the writer to flush the buffer.
	2514	*
	2515	* This is currently undocumented since it's not entirely useful on its own.
	2516	*/
	2517
	2518	#define mpack_writer_init_stack(writer) \
	2519	mpack_writer_init_stack_line(__LINE__, (writer))
	2520
	2521	/** @endcond */
	2522
	2523	/**
	2524	* Cleans up the MPack writer, flushing and closing the underlying stream,
	2525	* if any. Returns the final error state of the writer.
	2526	*
	2527	* No flushing is performed if the writer is in an error state. The attached
	2528	* teardown function is called whether or not the writer is in an error state.
	2529	*
	2530	* This will assert in tracking mode if the writer is not in an error
	2531	* state and has any unclosed compound types. If you want to cancel
	2532	* writing in the middle of a document, you need to flag an error on
	2533	* the writer before destroying it (such as mpack_error_data).
	2534	*
	2535	* Note that a writer may raise an error and call your error handler during
	2536	* the final flush. It is safe to longjmp or throw out of this error handler,
	2537	* but if you do, the writer will not be destroyed, and the teardown function
	2538	* will not be called. You can still get the writer's error state, and you
	2539	* must call @ref mpack_writer_destroy() again. (The second call is guaranteed
	2540	* not to call your error handler again since the writer is already in an error
	2541	* state.)
	2542	*
	2543	* @see mpack_writer_set_error_handler
	2544	* @see mpack_writer_set_flush
	2545	* @see mpack_writer_set_teardown
	2546	* @see mpack_writer_flag_error
	2547	* @see mpack_error_data
	2548	*/
	2549	mpack_error_t mpack_writer_destroy(mpack_writer_t* writer);
	2550
	2551	/**
	2552	* @}
	2553	*/
	2554
	2555	/**
	2556	* @name Configuration
	2557	* @{
	2558	*/
	2559
	2560	#if MPACK_COMPATIBILITY
	2561	/**
	2562	* Sets the version of the MessagePack spec that will be generated.
	2563	*
	2564	* This can be used to interface with older libraries that do not support
	2565	* the newest MessagePack features (such as the @c str8 type.)
	2566	*
	2567	* @note This requires @ref MPACK_COMPATIBILITY.
	2568	*/
	2569	MPACK_INLINE void mpack_writer_set_version(mpack_writer_t* writer, mpack_version_t version) {
	2570	writer->version = version;
	2571	}
	2572	#endif
	2573
	2574	/**
	2575	* Sets the custom pointer to pass to the writer callbacks, such as flush
	2576	* or teardown.
	2577	*
	2578	* @param writer The MPack writer.
	2579	* @param context User data to pass to the writer callbacks.
	2580	*
	2581	* @see mpack_writer_context()
	2582	*/
	2583	MPACK_INLINE void mpack_writer_set_context(mpack_writer_t* writer, void* context) {
	2584	writer->context = context;
	2585	}
	2586
	2587	/**
	2588	* Returns the custom context for writer callbacks.
	2589	*
	2590	* @see mpack_writer_set_context
	2591	* @see mpack_writer_set_flush
	2592	*/
	2593	MPACK_INLINE void* mpack_writer_context(mpack_writer_t* writer) {
	2594	return writer->context;
	2595	}
	2596
	2597	/**
	2598	* Sets the flush function to write out the data when the buffer is full.
	2599	*
	2600	* If no flush function is used, trying to write past the end of the
	2601	* buffer will result in mpack_error_too_big.
	2602	*
	2603	* This should normally be used with mpack_writer_set_context() to register
	2604	* a custom pointer to pass to the flush function.
	2605	*
	2606	* @param writer The MPack writer.
	2607	* @param flush The function to write out data from the buffer.
	2608	*
	2609	* @see mpack_writer_context()
	2610	*/
	2611	void mpack_writer_set_flush(mpack_writer_t* writer, mpack_writer_flush_t flush);
	2612
	2613	/**
	2614	* Sets the error function to call when an error is flagged on the writer.
	2615	*
	2616	* This should normally be used with mpack_writer_set_context() to register
	2617	* a custom pointer to pass to the error function.
	2618	*
	2619	* See the definition of mpack_writer_error_t for more information about
	2620	* what you can do from an error callback.
	2621	*
	2622	* @see mpack_writer_error_t
	2623	* @param writer The MPack writer.
	2624	* @param error_fn The function to call when an error is flagged on the writer.
	2625	*/
	2626	MPACK_INLINE void mpack_writer_set_error_handler(mpack_writer_t* writer, mpack_writer_error_t error_fn) {
	2627	writer->error_fn = error_fn;
	2628	}
	2629
	2630	/**
	2631	* Sets the teardown function to call when the writer is destroyed.
	2632	*
	2633	* This should normally be used with mpack_writer_set_context() to register
	2634	* a custom pointer to pass to the teardown function.
	2635	*
	2636	* @param writer The MPack writer.
	2637	* @param teardown The function to call when the writer is destroyed.
	2638	*/
	2639	MPACK_INLINE void mpack_writer_set_teardown(mpack_writer_t* writer, mpack_writer_teardown_t teardown) {
	2640	writer->teardown = teardown;
	2641	}
	2642
	2643	/**
	2644	* @}
	2645	*/
	2646
	2647	/**
	2648	* @name Core Writer Functions
	2649	* @{
	2650	*/
	2651
	2652	/**
	2653	* Flushes any buffered data to the underlying stream.
	2654	*
	2655	* If write tracking is enabled, this will break and flag @ref
	2656	* mpack_error_bug if the writer has any open compound types, ensuring
	2657	* that no compound types are still open. This prevents a "missing
	2658	* finish" bug from causing a never-ending message.
	2659	*
	2660	* If the writer is connected to a socket and you are keeping it open,
	2661	* you will want to call this after writing a message (or set of
	2662	* messages) so that the data is actually sent.
	2663	*
	2664	* It is not necessary to call this if you are not keeping the writer
	2665	* open afterwards. You can just call `mpack_writer_destroy()`, and it
	2666	* will flush before cleaning up.
	2667	*
	2668	* This will assert if no flush function is assigned to the writer.
	2669	*/
	2670	void mpack_writer_flush_message(mpack_writer_t* writer);
	2671
	2672	/**
	2673	* Returns the number of bytes currently stored in the buffer. This
	2674	* may be less than the total number of bytes written if bytes have
	2675	* been flushed to an underlying stream.
	2676	*/
	2677	MPACK_INLINE size_t mpack_writer_buffer_used(mpack_writer_t* writer) {
	2678	return (size_t)(writer->current - writer->buffer);
	2679	}
	2680
	2681	/**
	2682	* Returns the amount of space left in the buffer. This may be reset
	2683	* after a write if bytes are flushed to an underlying stream.
	2684	*/
	2685	MPACK_INLINE size_t mpack_writer_buffer_left(mpack_writer_t* writer) {
	2686	return (size_t)(writer->end - writer->current);
	2687	}
	2688
	2689	/**
	2690	* Returns the (current) size of the buffer. This may change after a write if
	2691	* the flush callback changes the buffer.
	2692	*/
	2693	MPACK_INLINE size_t mpack_writer_buffer_size(mpack_writer_t* writer) {
	2694	return (size_t)(writer->end - writer->buffer);
	2695	}
	2696
	2697	/**
	2698	* Places the writer in the given error state, calling the error callback if one
	2699	* is set.
	2700	*
	2701	* This allows you to externally flag errors, for example if you are validating
	2702	* data as you write it, or if you want to cancel writing in the middle of a
	2703	* document. (The writer will assert if you try to destroy it without error and
	2704	* with unclosed compound types. In this case you should flag mpack_error_data
	2705	* before destroying it.)
	2706	*
	2707	* If the writer is already in an error state, this call is ignored and no
	2708	* error callback is called.
	2709	*
	2710	* @see mpack_writer_destroy
	2711	* @see mpack_error_data
	2712	*/
	2713	void mpack_writer_flag_error(mpack_writer_t* writer, mpack_error_t error);
	2714
	2715	/**
	2716	* Queries the error state of the MPack writer.
	2717	*
	2718	* If a writer is in an error state, you should discard all data since the
	2719	* last time the error flag was checked. The error flag cannot be cleared.
	2720	*/
	2721	MPACK_INLINE mpack_error_t mpack_writer_error(mpack_writer_t* writer) {
	2722	return writer->error;
	2723	}
	2724
	2725	/**
	2726	* Writes a MessagePack object header (an MPack Tag.)
	2727	*
	2728	* If the value is a map, array, string, binary or extension type, the
	2729	* containing elements or bytes must be written separately and the
	2730	* appropriate finish function must be called (as though one of the
	2731	* mpack_start_*() functions was called.)
	2732	*
	2733	* @see mpack_write_bytes()
	2734	* @see mpack_finish_map()
	2735	* @see mpack_finish_array()
	2736	* @see mpack_finish_str()
	2737	* @see mpack_finish_bin()
	2738	* @see mpack_finish_ext()
	2739	* @see mpack_finish_type()
	2740	*/
	2741	void mpack_write_tag(mpack_writer_t* writer, mpack_tag_t tag);
	2742
	2743	/**
	2744	* @}
	2745	*/
	2746
	2747	/**
	2748	* @name Integers
	2749	* @{
	2750	*/
	2751
	2752	/** Writes an 8-bit integer in the most efficient packing available. */
	2753	void mpack_write_i8(mpack_writer_t* writer, int8_t value);
	2754
	2755	/** Writes a 16-bit integer in the most efficient packing available. */
	2756	void mpack_write_i16(mpack_writer_t* writer, int16_t value);
	2757
	2758	/** Writes a 32-bit integer in the most efficient packing available. */
	2759	void mpack_write_i32(mpack_writer_t* writer, int32_t value);
	2760
	2761	/** Writes a 64-bit integer in the most efficient packing available. */
	2762	void mpack_write_i64(mpack_writer_t* writer, int64_t value);
	2763
	2764	/** Writes an integer in the most efficient packing available. */
	2765	MPACK_INLINE void mpack_write_int(mpack_writer_t* writer, int64_t value) {
	2766	mpack_write_i64(writer, value);
	2767	}
	2768
	2769	/** Writes an 8-bit unsigned integer in the most efficient packing available. */
	2770	void mpack_write_u8(mpack_writer_t* writer, uint8_t value);
	2771
	2772	/** Writes an 16-bit unsigned integer in the most efficient packing available. */
	2773	void mpack_write_u16(mpack_writer_t* writer, uint16_t value);
	2774
	2775	/** Writes an 32-bit unsigned integer in the most efficient packing available. */
	2776	void mpack_write_u32(mpack_writer_t* writer, uint32_t value);
	2777
	2778	/** Writes an 64-bit unsigned integer in the most efficient packing available. */
	2779	void mpack_write_u64(mpack_writer_t* writer, uint64_t value);
	2780
	2781	/** Writes an unsigned integer in the most efficient packing available. */
	2782	MPACK_INLINE void mpack_write_uint(mpack_writer_t* writer, uint64_t value) {
	2783	mpack_write_u64(writer, value);
	2784	}
	2785
	2786	/**
	2787	* @}
	2788	*/
	2789
	2790	/**
	2791	* @name Other Basic Types
	2792	* @{
	2793	*/
	2794
	2795	/** Writes a float. */
	2796	void mpack_write_float(mpack_writer_t* writer, float value);
	2797
	2798	/** Writes a double. */
	2799	void mpack_write_double(mpack_writer_t* writer, double value);
	2800
	2801	/** Writes a boolean. */
	2802	void mpack_write_bool(mpack_writer_t* writer, bool value);
	2803
	2804	/** Writes a boolean with value true. */
	2805	void mpack_write_true(mpack_writer_t* writer);
	2806
	2807	/** Writes a boolean with value false. */
	2808	void mpack_write_false(mpack_writer_t* writer);
	2809
	2810	/** Writes a nil. */
	2811	void mpack_write_nil(mpack_writer_t* writer);
	2812
	2813	/** Write a pre-encoded messagepack object */
	2814	void mpack_write_object_bytes(mpack_writer_t* writer, const char* data, size_t bytes);
	2815
	2816	#if MPACK_EXTENSIONS
	2817	/**
	2818	* Writes a timestamp.
	2819	*
	2820	* @note This requires @ref MPACK_EXTENSIONS.
	2821	*
	2822	* @param writer The writer
	2823	* @param seconds The (signed) number of seconds since 1970-01-01T00:00:00Z.
	2824	* @param nanoseconds The additional number of nanoseconds from 0 to 999,999,999 inclusive.
	2825	*/
	2826	void mpack_write_timestamp(mpack_writer_t* writer, int64_t seconds, uint32_t nanoseconds);
	2827
	2828	/**
	2829	* Writes a timestamp with the given number of seconds (and zero nanoseconds).
	2830	*
	2831	* @note This requires @ref MPACK_EXTENSIONS.
	2832	*
	2833	* @param writer The writer
	2834	* @param seconds The (signed) number of seconds since 1970-01-01T00:00:00Z.
	2835	*/
	2836	MPACK_INLINE void mpack_write_timestamp_seconds(mpack_writer_t* writer, int64_t seconds) {
	2837	mpack_write_timestamp(writer, seconds, 0);
	2838	}
	2839
	2840	/**
	2841	* Writes a timestamp.
	2842	*
	2843	* @note This requires @ref MPACK_EXTENSIONS.
	2844	*/
	2845	MPACK_INLINE void mpack_write_timestamp_struct(mpack_writer_t* writer, mpack_timestamp_t timestamp) {
	2846	mpack_write_timestamp(writer, timestamp.seconds, timestamp.nanoseconds);
	2847	}
	2848	#endif
	2849
	2850	/**
	2851	* @}
	2852	*/
	2853
	2854	/**
	2855	* @name Map and Array Functions
	2856	* @{
	2857	*/
	2858
	2859	/**
	2860	* Opens an array.
	2861	*
	2862	* `count` elements must follow, and mpack_finish_array() must be called
	2863	* when done.
	2864	*
	2865	* @see mpack_finish_array()
	2866	*/
	2867	void mpack_start_array(mpack_writer_t* writer, uint32_t count);
	2868
	2869	/**
	2870	* Opens a map.
	2871	*
	2872	* `count * 2` elements must follow, and mpack_finish_map() must be called
	2873	* when done.
	2874	*
	2875	* Remember that while map elements in MessagePack are implicitly ordered,
	2876	* they are not ordered in JSON. If you need elements to be read back
	2877	* in the order they are written, consider use an array instead.
	2878	*
	2879	* @see mpack_finish_map()
	2880	*/
	2881	void mpack_start_map(mpack_writer_t* writer, uint32_t count);
	2882
	2883	/**
	2884	* Finishes writing an array.
	2885	*
	2886	* This should be called only after a corresponding call to mpack_start_array()
	2887	* and after the array contents are written.
	2888	*
	2889	* This will track writes to ensure that the correct number of elements are written.
	2890	*
	2891	* @see mpack_start_array()
	2892	*/
	2893	MPACK_INLINE void mpack_finish_array(mpack_writer_t* writer) {
	2894	mpack_writer_track_pop(writer, mpack_type_array);
	2895	}
	2896
	2897	/**
	2898	* Finishes writing a map.
	2899	*
	2900	* This should be called only after a corresponding call to mpack_start_map()
	2901	* and after the map contents are written.
	2902	*
	2903	* This will track writes to ensure that the correct number of elements are written.
	2904	*
	2905	* @see mpack_start_map()
	2906	*/
	2907	MPACK_INLINE void mpack_finish_map(mpack_writer_t* writer) {
	2908	mpack_writer_track_pop(writer, mpack_type_map);
	2909	}
	2910
	2911	/**
	2912	* @}
	2913	*/
	2914
	2915	/**
	2916	* @name Data Helpers
	2917	* @{
	2918	*/
	2919
	2920	/**
	2921	* Writes a string.
	2922	*
	2923	* To stream a string in chunks, use mpack_start_str() instead.
	2924	*
	2925	* MPack does not care about the underlying encoding, but UTF-8 is highly
	2926	* recommended, especially for compatibility with JSON. You should consider
	2927	* calling mpack_write_utf8() instead, especially if you will be reading
	2928	* it back as UTF-8.
	2929	*
	2930	* You should not call mpack_finish_str() after calling this; this
	2931	* performs both start and finish.
	2932	*/
	2933	void mpack_write_str(mpack_writer_t* writer, const char* str, uint32_t length);
	2934
	2935	/**
	2936	* Writes a string, ensuring that it is valid UTF-8.
	2937	*
	2938	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	2939	* WTF-8. Only pure UTF-8 is allowed.
	2940	*
	2941	* You should not call mpack_finish_str() after calling this; this
	2942	* performs both start and finish.
	2943	*
	2944	* @throws mpack_error_invalid if the string is not valid UTF-8
	2945	*/
	2946	void mpack_write_utf8(mpack_writer_t* writer, const char* str, uint32_t length);
	2947
	2948	/**
	2949	* Writes a null-terminated string. (The null-terminator is not written.)
	2950	*
	2951	* MPack does not care about the underlying encoding, but UTF-8 is highly
	2952	* recommended, especially for compatibility with JSON. You should consider
	2953	* calling mpack_write_utf8_cstr() instead, especially if you will be reading
	2954	* it back as UTF-8.
	2955	*
	2956	* You should not call mpack_finish_str() after calling this; this
	2957	* performs both start and finish.
	2958	*/
	2959	void mpack_write_cstr(mpack_writer_t* writer, const char* cstr);
	2960
	2961	/**
	2962	* Writes a null-terminated string, or a nil node if the given cstr pointer
	2963	* is NULL. (The null-terminator is not written.)
	2964	*
	2965	* MPack does not care about the underlying encoding, but UTF-8 is highly
	2966	* recommended, especially for compatibility with JSON. You should consider
	2967	* calling mpack_write_utf8_cstr_or_nil() instead, especially if you will
	2968	* be reading it back as UTF-8.
	2969	*
	2970	* You should not call mpack_finish_str() after calling this; this
	2971	* performs both start and finish.
	2972	*/
	2973	void mpack_write_cstr_or_nil(mpack_writer_t* writer, const char* cstr);
	2974
	2975	/**
	2976	* Writes a null-terminated string, ensuring that it is valid UTF-8. (The
	2977	* null-terminator is not written.)
	2978	*
	2979	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	2980	* WTF-8. Only pure UTF-8 is allowed.
	2981	*
	2982	* You should not call mpack_finish_str() after calling this; this
	2983	* performs both start and finish.
	2984	*
	2985	* @throws mpack_error_invalid if the string is not valid UTF-8
	2986	*/
	2987	void mpack_write_utf8_cstr(mpack_writer_t* writer, const char* cstr);
	2988
	2989	/**
	2990	* Writes a null-terminated string ensuring that it is valid UTF-8, or
	2991	* writes nil if the given cstr pointer is NULL. (The null-terminator
	2992	* is not written.)
	2993	*
	2994	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	2995	* WTF-8. Only pure UTF-8 is allowed.
	2996	*
	2997	* You should not call mpack_finish_str() after calling this; this
	2998	* performs both start and finish.
	2999	*
	3000	* @throws mpack_error_invalid if the string is not valid UTF-8
	3001	*/
	3002	void mpack_write_utf8_cstr_or_nil(mpack_writer_t* writer, const char* cstr);
	3003
	3004	/**
	3005	* Writes a binary blob.
	3006	*
	3007	* To stream a binary blob in chunks, use mpack_start_bin() instead.
	3008	*
	3009	* You should not call mpack_finish_bin() after calling this; this
	3010	* performs both start and finish.
	3011	*/
	3012	void mpack_write_bin(mpack_writer_t* writer, const char* data, uint32_t count);
	3013
	3014	#if MPACK_EXTENSIONS
	3015	/**
	3016	* Writes an extension type.
	3017	*
	3018	* To stream an extension blob in chunks, use mpack_start_ext() instead.
	3019	*
	3020	* Extension types [0, 127] are available for application-specific types. Extension
	3021	* types [-128, -1] are reserved for future extensions of MessagePack.
	3022	*
	3023	* You should not call mpack_finish_ext() after calling this; this
	3024	* performs both start and finish.
	3025	*
	3026	* @note This requires @ref MPACK_EXTENSIONS.
	3027	*/
	3028	void mpack_write_ext(mpack_writer_t* writer, int8_t exttype, const char* data, uint32_t count);
	3029	#endif
	3030
	3031	/**
	3032	* @}
	3033	*/
	3034
	3035	/**
	3036	* @name Chunked Data Functions
	3037	* @{
	3038	*/
	3039
	3040	/**
	3041	* Opens a string. `count` bytes should be written with calls to
	3042	* mpack_write_bytes(), and mpack_finish_str() should be called
	3043	* when done.
	3044	*
	3045	* To write an entire string at once, use mpack_write_str() or
	3046	* mpack_write_cstr() instead.
	3047	*
	3048	* MPack does not care about the underlying encoding, but UTF-8 is highly
	3049	* recommended, especially for compatibility with JSON.
	3050	*/
	3051	void mpack_start_str(mpack_writer_t* writer, uint32_t count);
	3052
	3053	/**
	3054	* Opens a binary blob. `count` bytes should be written with calls to
	3055	* mpack_write_bytes(), and mpack_finish_bin() should be called
	3056	* when done.
	3057	*/
	3058	void mpack_start_bin(mpack_writer_t* writer, uint32_t count);
	3059
	3060	#if MPACK_EXTENSIONS
	3061	/**
	3062	* Opens an extension type. `count` bytes should be written with calls
	3063	* to mpack_write_bytes(), and mpack_finish_ext() should be called
	3064	* when done.
	3065	*
	3066	* Extension types [0, 127] are available for application-specific types. Extension
	3067	* types [-128, -1] are reserved for future extensions of MessagePack.
	3068	*
	3069	* @note This requires @ref MPACK_EXTENSIONS.
	3070	*/
	3071	void mpack_start_ext(mpack_writer_t* writer, int8_t exttype, uint32_t count);
	3072	#endif
	3073
	3074	/**
	3075	* Writes a portion of bytes for a string, binary blob or extension type which
	3076	* was opened by mpack_write_tag() or one of the mpack_start_*() functions.
	3077	*
	3078	* This can be called multiple times to write the data in chunks, as long as
	3079	* the total amount of bytes written matches the count given when the compound
	3080	* type was started.
	3081	*
	3082	* The corresponding mpack_finish_*() function must be called when done.
	3083	*
	3084	* To write an entire string, binary blob or extension type at
	3085	* once, use one of the mpack_write_*() functions instead.
	3086	*
	3087	* @see mpack_write_tag()
	3088	* @see mpack_start_str()
	3089	* @see mpack_start_bin()
	3090	* @see mpack_start_ext()
	3091	* @see mpack_finish_str()
	3092	* @see mpack_finish_bin()
	3093	* @see mpack_finish_ext()
	3094	* @see mpack_finish_type()
	3095	*/
	3096	void mpack_write_bytes(mpack_writer_t* writer, const char* data, size_t count);
	3097
	3098	/**
	3099	* Finishes writing a string.
	3100	*
	3101	* This should be called only after a corresponding call to mpack_start_str()
	3102	* and after the string bytes are written with mpack_write_bytes().
	3103	*
	3104	* This will track writes to ensure that the correct number of elements are written.
	3105	*
	3106	* @see mpack_start_str()
	3107	* @see mpack_write_bytes()
	3108	*/
	3109	MPACK_INLINE void mpack_finish_str(mpack_writer_t* writer) {
	3110	mpack_writer_track_pop(writer, mpack_type_str);
	3111	}
	3112
	3113	/**
	3114	* Finishes writing a binary blob.
	3115	*
	3116	* This should be called only after a corresponding call to mpack_start_bin()
	3117	* and after the binary bytes are written with mpack_write_bytes().
	3118	*
	3119	* This will track writes to ensure that the correct number of bytes are written.
	3120	*
	3121	* @see mpack_start_bin()
	3122	* @see mpack_write_bytes()
	3123	*/
	3124	MPACK_INLINE void mpack_finish_bin(mpack_writer_t* writer) {
	3125	mpack_writer_track_pop(writer, mpack_type_bin);
	3126	}
	3127
	3128	#if MPACK_EXTENSIONS
	3129	/**
	3130	* Finishes writing an extended type binary data blob.
	3131	*
	3132	* This should be called only after a corresponding call to mpack_start_bin()
	3133	* and after the binary bytes are written with mpack_write_bytes().
	3134	*
	3135	* This will track writes to ensure that the correct number of bytes are written.
	3136	*
	3137	* @note This requires @ref MPACK_EXTENSIONS.
	3138	*
	3139	* @see mpack_start_ext()
	3140	* @see mpack_write_bytes()
	3141	*/
	3142	MPACK_INLINE void mpack_finish_ext(mpack_writer_t* writer) {
	3143	mpack_writer_track_pop(writer, mpack_type_ext);
	3144	}
	3145	#endif
	3146
	3147	/**
	3148	* Finishes writing the given compound type.
	3149	*
	3150	* This will track writes to ensure that the correct number of elements
	3151	* or bytes are written.
	3152	*
	3153	* This can be called with the appropriate type instead the corresponding
	3154	* mpack_finish_*() function if you want to finish a dynamic type.
	3155	*/
	3156	MPACK_INLINE void mpack_finish_type(mpack_writer_t* writer, mpack_type_t type) {
	3157	mpack_writer_track_pop(writer, type);
	3158	}
	3159
	3160	/**
	3161	* @}
	3162	*/
	3163
	3164	#if MPACK_WRITER && MPACK_HAS_GENERIC && !defined(__cplusplus)
	3165
	3166	/**
	3167	* @name Type-Generic Writers
	3168	* @{
	3169	*/
	3170
	3171	/**
	3172	* @def mpack_write(writer, value)
	3173	*
	3174	* Type-generic writer for primitive types.
	3175	*
	3176	* The compiler will dispatch to an appropriate write function based
	3177	* on the type of the @a value parameter.
	3178	*
	3179	* @note This requires C11 `_Generic` support. (A set of inline overloads
	3180	* are used in C++ to provide the same functionality.)
	3181	*
	3182	* @warning In C11, the indentifiers `true`, `false` and `NULL` are
	3183	* all of type `int`, not `bool` or `void*`! They will emit unexpected
	3184	* types when passed uncast, so be careful when using them.
	3185	*/
	3186	#define mpack_write(writer, value) \
	3187	_Generic(((void)0, value), \
	3188	int8_t: mpack_write_i8, \
	3189	int16_t: mpack_write_i16, \
	3190	int32_t: mpack_write_i32, \
	3191	int64_t: mpack_write_i64, \
	3192	uint8_t: mpack_write_u8, \
	3193	uint16_t: mpack_write_u16, \
	3194	uint32_t: mpack_write_u32, \
	3195	uint64_t: mpack_write_u64, \
	3196	bool: mpack_write_bool, \
	3197	float: mpack_write_float, \
	3198	double: mpack_write_double, \
	3199	char *: mpack_write_cstr_or_nil, \
	3200	const char *: mpack_write_cstr_or_nil \
	3201	)(writer, value)
	3202
	3203	/**
	3204	* @def mpack_write_kv(writer, key, value)
	3205	*
	3206	* Type-generic writer for key-value pairs of null-terminated string
	3207	* keys and primitive values.
	3208	*
	3209	* @warning @a writer may be evaluated multiple times.
	3210	*
	3211	* @warning In C11, the indentifiers `true`, `false` and `NULL` are
	3212	* all of type `int`, not `bool` or `void*`! They will emit unexpected
	3213	* types when passed uncast, so be careful when using them.
	3214	*
	3215	* @param writer The writer.
	3216	* @param key A null-terminated C string.
	3217	* @param value A primitive type supported by mpack_write().
	3218	*/
	3219	#define mpack_write_kv(writer, key, value) do { \
	3220	mpack_write_cstr(writer, key); \
	3221	mpack_write(writer, value); \
	3222	} while (0)
	3223
	3224	/**
	3225	* @}
	3226	*/
	3227
	3228	#endif
	3229
	3230	/**
	3231	* @}
	3232	*/
	3233
	3234	#endif
	3235
	3236	MPACK_HEADER_END
	3237
	3238	#if defined(__cplusplus) \|\| defined(MPACK_DOXYGEN)
	3239
	3240	/*
	3241	* C++ generic writers for primitive values
	3242	*
	3243	* These currently sit outside of MPACK_HEADER_END because it defines
	3244	* extern "C". They'll be moved to a C++-specific header soon.
	3245	*/
	3246
	3247	#ifdef MPACK_DOXYGEN
	3248	#undef mpack_write
	3249	#undef mpack_write_kv
	3250	#endif
	3251
	3252	MPACK_INLINE void mpack_write(mpack_writer_t* writer, int8_t value) {
	3253	mpack_write_i8(writer, value);
	3254	}
	3255
	3256	MPACK_INLINE void mpack_write(mpack_writer_t* writer, int16_t value) {
	3257	mpack_write_i16(writer, value);
	3258	}
	3259
	3260	MPACK_INLINE void mpack_write(mpack_writer_t* writer, int32_t value) {
	3261	mpack_write_i32(writer, value);
	3262	}
	3263
	3264	MPACK_INLINE void mpack_write(mpack_writer_t* writer, int64_t value) {
	3265	mpack_write_i64(writer, value);
	3266	}
	3267
	3268	MPACK_INLINE void mpack_write(mpack_writer_t* writer, uint8_t value) {
	3269	mpack_write_u8(writer, value);
	3270	}
	3271
	3272	MPACK_INLINE void mpack_write(mpack_writer_t* writer, uint16_t value) {
	3273	mpack_write_u16(writer, value);
	3274	}
	3275
	3276	MPACK_INLINE void mpack_write(mpack_writer_t* writer, uint32_t value) {
	3277	mpack_write_u32(writer, value);
	3278	}
	3279
	3280	MPACK_INLINE void mpack_write(mpack_writer_t* writer, uint64_t value) {
	3281	mpack_write_u64(writer, value);
	3282	}
	3283
	3284	MPACK_INLINE void mpack_write(mpack_writer_t* writer, bool value) {
	3285	mpack_write_bool(writer, value);
	3286	}
	3287
	3288	MPACK_INLINE void mpack_write(mpack_writer_t* writer, float value) {
	3289	mpack_write_float(writer, value);
	3290	}
	3291
	3292	MPACK_INLINE void mpack_write(mpack_writer_t* writer, double value) {
	3293	mpack_write_double(writer, value);
	3294	}
	3295
	3296	MPACK_INLINE void mpack_write(mpack_writer_t* writer, char *value) {
	3297	mpack_write_cstr_or_nil(writer, value);
	3298	}
	3299
	3300	MPACK_INLINE void mpack_write(mpack_writer_t* writer, const char *value) {
	3301	mpack_write_cstr_or_nil(writer, value);
	3302	}
	3303
	3304	/* C++ generic write for key-value pairs */
	3305
	3306	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, int8_t value) {
	3307	mpack_write_cstr(writer, key);
	3308	mpack_write_i8(writer, value);
	3309	}
	3310
	3311	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, int16_t value) {
	3312	mpack_write_cstr(writer, key);
	3313	mpack_write_i16(writer, value);
	3314	}
	3315
	3316	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, int32_t value) {
	3317	mpack_write_cstr(writer, key);
	3318	mpack_write_i32(writer, value);
	3319	}
	3320
	3321	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, int64_t value) {
	3322	mpack_write_cstr(writer, key);
	3323	mpack_write_i64(writer, value);
	3324	}
	3325
	3326	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, uint8_t value) {
	3327	mpack_write_cstr(writer, key);
	3328	mpack_write_u8(writer, value);
	3329	}
	3330
	3331	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, uint16_t value) {
	3332	mpack_write_cstr(writer, key);
	3333	mpack_write_u16(writer, value);
	3334	}
	3335
	3336	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, uint32_t value) {
	3337	mpack_write_cstr(writer, key);
	3338	mpack_write_u32(writer, value);
	3339	}
	3340
	3341	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, uint64_t value) {
	3342	mpack_write_cstr(writer, key);
	3343	mpack_write_u64(writer, value);
	3344	}
	3345
	3346	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, bool value) {
	3347	mpack_write_cstr(writer, key);
	3348	mpack_write_bool(writer, value);
	3349	}
	3350
	3351	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, float value) {
	3352	mpack_write_cstr(writer, key);
	3353	mpack_write_float(writer, value);
	3354	}
	3355
	3356	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char *key, double value) {
	3357	mpack_write_cstr(writer, key);
	3358	mpack_write_double(writer, value);
	3359	}
	3360
	3361	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char key, char value) {
	3362	mpack_write_cstr(writer, key);
	3363	mpack_write_cstr_or_nil(writer, value);
	3364	}
	3365
	3366	MPACK_INLINE void mpack_write_kv(mpack_writer_t* writer, const char key, const char value) {
	3367	mpack_write_cstr(writer, key);
	3368	mpack_write_cstr_or_nil(writer, value);
	3369	}
	3370	#endif /* __cplusplus */
	3371
	3372	#endif
	3373
	3374	/* mpack/mpack-reader.h.h */
	3375
	3376	/**
	3377	* @file
	3378	*
	3379	* Declares the core MPack Tag Reader.
	3380	*/
	3381
	3382	#ifndef MPACK_READER_H
	3383	#define MPACK_READER_H 1
	3384
	3385	/* #include "mpack-common.h" */
	3386
	3387	MPACK_HEADER_START
	3388
	3389	#if MPACK_READER
	3390
	3391	#if MPACK_READ_TRACKING
	3392	struct mpack_track_t;
	3393	#endif
	3394
	3395	// The denominator to determine whether a read is a small
	3396	// fraction of the buffer size.
	3397	#define MPACK_READER_SMALL_FRACTION_DENOMINATOR 32
	3398
	3399	/**
	3400	* @defgroup reader Reader API
	3401	*
	3402	* The MPack Reader API contains functions for imperatively reading dynamically
	3403	* typed data from a MessagePack stream.
	3404	*
	3405	* See @ref docs/reader.md for examples.
	3406	*
	3407	* @note If you are not writing code for an embedded device (or otherwise do
	3408	* not need maximum performance with minimal memory usage), you should not use
	3409	* this. You probably want to use the @link node Node API@endlink instead.
	3410	*
	3411	* This forms the basis of the @link expect Expect API@endlink, which can be
	3412	* used to interpret the stream of elements in expected types and value ranges.
	3413	*
	3414	* @{
	3415	*/
	3416
	3417	/**
	3418	* @def MPACK_READER_MINIMUM_BUFFER_SIZE
	3419	*
	3420	* The minimum buffer size for a reader with a fill function.
	3421	*/
	3422	#define MPACK_READER_MINIMUM_BUFFER_SIZE 32
	3423
	3424	/**
	3425	* A buffered MessagePack decoder.
	3426	*
	3427	* The decoder wraps an existing buffer and, optionally, a fill function.
	3428	* This allows efficiently decoding data from existing memory buffers, files,
	3429	* streams, etc.
	3430	*
	3431	* All read operations are synchronous; they will block until the
	3432	* requested data is fully read, or an error occurs.
	3433	*
	3434	* This structure is opaque; its fields should not be accessed outside
	3435	* of MPack.
	3436	*/
	3437	typedef struct mpack_reader_t mpack_reader_t;
	3438
	3439	/**
	3440	* The MPack reader's fill function. It should fill the buffer with at
	3441	* least one byte and at most the given @c count, returning the number
	3442	* of bytes written to the buffer.
	3443	*
	3444	* In case of error, it should flag an appropriate error on the reader
	3445	* (usually @ref mpack_error_io), or simply return zero. If zero is
	3446	* returned, mpack_error_io is raised.
	3447	*
	3448	* @note When reading from a stream, you should only copy and return
	3449	* the bytes that are immediately available. It is always safe to return
	3450	* less than the requested count as long as some non-zero number of bytes
	3451	* are read; if more bytes are needed, the read function will simply be
	3452	* called again.
	3453	*
	3454	* @see mpack_reader_context()
	3455	*/
	3456	typedef size_t (mpack_reader_fill_t)(mpack_reader_t reader, char* buffer, size_t count);
	3457
	3458	/**
	3459	* The MPack reader's skip function. It should discard the given number
	3460	* of bytes from the source (for example by seeking forward.)
	3461	*
	3462	* In case of error, it should flag an appropriate error on the reader.
	3463	*
	3464	* @see mpack_reader_context()
	3465	*/
	3466	typedef void (mpack_reader_skip_t)(mpack_reader_t reader, size_t count);
	3467
	3468	/**
	3469	* An error handler function to be called when an error is flagged on
	3470	* the reader.
	3471	*
	3472	* The error handler will only be called once on the first error flagged;
	3473	* any subsequent reads and errors are ignored, and the reader is
	3474	* permanently in that error state.
	3475	*
	3476	* MPack is safe against non-local jumps out of error handler callbacks.
	3477	* This means you are allowed to longjmp or throw an exception (in C++,
	3478	* Objective-C, or with SEH) out of this callback.
	3479	*
	3480	* Bear in mind when using longjmp that local non-volatile variables that
	3481	* have changed are undefined when setjmp() returns, so you can't put the
	3482	* reader on the stack in the same activation frame as the setjmp without
	3483	* declaring it volatile.
	3484	*
	3485	* You must still eventually destroy the reader. It is not destroyed
	3486	* automatically when an error is flagged. It is safe to destroy the
	3487	* reader within this error callback, but you will either need to perform
	3488	* a non-local jump, or store something in your context to identify
	3489	* that the reader is destroyed since any future accesses to it cause
	3490	* undefined behavior.
	3491	*/
	3492	typedef void (mpack_reader_error_t)(mpack_reader_t reader, mpack_error_t error);
	3493
	3494	/**
	3495	* A teardown function to be called when the reader is destroyed.
	3496	*/
	3497	typedef void (mpack_reader_teardown_t)(mpack_reader_t reader);
	3498
	3499	/* Hide internals from documentation */
	3500	/** @cond */
	3501
	3502	struct mpack_reader_t {
	3503	void* context; /* Context for reader callbacks */
	3504	mpack_reader_fill_t fill; /* Function to read bytes into the buffer */
	3505	mpack_reader_error_t error_fn; /* Function to call on error */
	3506	mpack_reader_teardown_t teardown; /* Function to teardown the context on destroy */
	3507	mpack_reader_skip_t skip; /* Function to skip bytes from the source */
	3508
	3509	char* buffer; /* Writeable byte buffer */
	3510	size_t size; /* Size of the buffer */
	3511
	3512	const char* data; /* Current data pointer (in the buffer, if it is used) */
	3513	const char* end; /* The end of available data (in the buffer, if it is used) */
	3514
	3515	mpack_error_t error; /* Error state */
	3516
	3517	#if MPACK_READ_TRACKING
	3518	mpack_track_t track; /* Stack of map/array/str/bin/ext reads */
	3519	#endif
	3520	};
	3521
	3522	/** @endcond */
	3523
	3524	/**
	3525	* @name Lifecycle Functions
	3526	* @{
	3527	*/
	3528
	3529	/**
	3530	* Initializes an MPack reader with the given buffer. The reader does
	3531	* not assume ownership of the buffer, but the buffer must be writeable
	3532	* if a fill function will be used to refill it.
	3533	*
	3534	* @param reader The MPack reader.
	3535	* @param buffer The buffer with which to read MessagePack data.
	3536	* @param size The size of the buffer.
	3537	* @param count The number of bytes already in the buffer.
	3538	*/
	3539	void mpack_reader_init(mpack_reader_t* reader, char* buffer, size_t size, size_t count);
	3540
	3541	/**
	3542	* Initializes an MPack reader directly into an error state. Use this if you
	3543	* are writing a wrapper to mpack_reader_init() which can fail its setup.
	3544	*/
	3545	void mpack_reader_init_error(mpack_reader_t* reader, mpack_error_t error);
	3546
	3547	/**
	3548	* Initializes an MPack reader to parse a pre-loaded contiguous chunk of data. The
	3549	* reader does not assume ownership of the data.
	3550	*
	3551	* @param reader The MPack reader.
	3552	* @param data The data to parse.
	3553	* @param count The number of bytes pointed to by data.
	3554	*/
	3555	void mpack_reader_init_data(mpack_reader_t* reader, const char* data, size_t count);
	3556
	3557	#if MPACK_STDIO
	3558	/**
	3559	* Initializes an MPack reader that reads from a file.
	3560	*
	3561	* The file will be automatically opened and closed by the reader.
	3562	*/
	3563	void mpack_reader_init_filename(mpack_reader_t* reader, const char* filename);
	3564
	3565	/**
	3566	* Deprecated.
	3567	*
	3568	* \deprecated Renamed to mpack_reader_init_filename().
	3569	*/
	3570	MPACK_INLINE void mpack_reader_init_file(mpack_reader_t* reader, const char* filename) {
	3571	mpack_reader_init_filename(reader, filename);
	3572	}
	3573
	3574	/**
	3575	* Initializes an MPack reader that reads from a libc FILE. This can be used to
	3576	* read from stdin, or from a file opened separately.
	3577	*
	3578	* @param reader The MPack reader.
	3579	* @param stdfile The FILE.
	3580	* @param close_when_done If true, fclose() will be called on the FILE when it
	3581	* is no longer needed. If false, the file will not be closed when
	3582	* reading is done.
	3583	*
	3584	* @warning The reader is buffered. It will read data in advance of parsing it,
	3585	* and it may read more data than it parsed. See mpack_reader_remaining() to
	3586	* access the extra data.
	3587	*/
	3588	void mpack_reader_init_stdfile(mpack_reader_t* reader, FILE* stdfile, bool close_when_done);
	3589	#endif
	3590
	3591	/**
	3592	* @def mpack_reader_init_stack(reader)
	3593	* @hideinitializer
	3594	*
	3595	* Initializes an MPack reader using stack space as a buffer. A fill function
	3596	* should be added to the reader to fill the buffer.
	3597	*
	3598	* @see mpack_reader_set_fill
	3599	*/
	3600
	3601	/** @cond */
	3602	#define mpack_reader_init_stack_line_ex(line, reader) \
	3603	char mpack_buf_##line[MPACK_STACK_SIZE]; \
	3604	mpack_reader_init((reader), mpack_buf_##line, sizeof(mpack_buf_##line), 0)
	3605
	3606	#define mpack_reader_init_stack_line(line, reader) \
	3607	mpack_reader_init_stack_line_ex(line, reader)
	3608	/** @endcond */
	3609
	3610	#define mpack_reader_init_stack(reader) \
	3611	mpack_reader_init_stack_line(__LINE__, (reader))
	3612
	3613	/**
	3614	* Cleans up the MPack reader, ensuring that all compound elements
	3615	* have been completely read. Returns the final error state of the
	3616	* reader.
	3617	*
	3618	* This will assert in tracking mode if the reader is not in an error
	3619	* state and has any incomplete reads. If you want to cancel reading
	3620	* in the middle of a document, you need to flag an error on the reader
	3621	* before destroying it (such as mpack_error_data).
	3622	*
	3623	* @see mpack_read_tag()
	3624	* @see mpack_reader_flag_error()
	3625	* @see mpack_error_data
	3626	*/
	3627	mpack_error_t mpack_reader_destroy(mpack_reader_t* reader);
	3628
	3629	/**
	3630	* @}
	3631	*/
	3632
	3633	/**
	3634	* @name Callbacks
	3635	* @{
	3636	*/
	3637
	3638	/**
	3639	* Sets the custom pointer to pass to the reader callbacks, such as fill
	3640	* or teardown.
	3641	*
	3642	* @param reader The MPack reader.
	3643	* @param context User data to pass to the reader callbacks.
	3644	*
	3645	* @see mpack_reader_context()
	3646	*/
	3647	MPACK_INLINE void mpack_reader_set_context(mpack_reader_t* reader, void* context) {
	3648	reader->context = context;
	3649	}
	3650
	3651	/**
	3652	* Returns the custom context for reader callbacks.
	3653	*
	3654	* @see mpack_reader_set_context
	3655	* @see mpack_reader_set_fill
	3656	* @see mpack_reader_set_skip
	3657	*/
	3658	MPACK_INLINE void* mpack_reader_context(mpack_reader_t* reader) {
	3659	return reader->context;
	3660	}
	3661
	3662	/**
	3663	* Sets the fill function to refill the data buffer when it runs out of data.
	3664	*
	3665	* If no fill function is used, truncated MessagePack data results in
	3666	* mpack_error_invalid (since the buffer is assumed to contain a
	3667	* complete MessagePack object.)
	3668	*
	3669	* If a fill function is used, truncated MessagePack data usually
	3670	* results in mpack_error_io (since the fill function fails to get
	3671	* the missing data.)
	3672	*
	3673	* This should normally be used with mpack_reader_set_context() to register
	3674	* a custom pointer to pass to the fill function.
	3675	*
	3676	* @param reader The MPack reader.
	3677	* @param fill The function to fetch additional data into the buffer.
	3678	*/
	3679	void mpack_reader_set_fill(mpack_reader_t* reader, mpack_reader_fill_t fill);
	3680
	3681	/**
	3682	* Sets the skip function to discard bytes from the source stream.
	3683	*
	3684	* It's not necessary to implement this function. If the stream is not
	3685	* seekable, don't set a skip callback. The reader will fall back to
	3686	* using the fill function instead.
	3687	*
	3688	* This should normally be used with mpack_reader_set_context() to register
	3689	* a custom pointer to pass to the skip function.
	3690	*
	3691	* The skip function is ignored in size-optimized builds to reduce code
	3692	* size. Data will be skipped with the fill function when necessary.
	3693	*
	3694	* @param reader The MPack reader.
	3695	* @param skip The function to discard bytes from the source stream.
	3696	*/
	3697	void mpack_reader_set_skip(mpack_reader_t* reader, mpack_reader_skip_t skip);
	3698
	3699	/**
	3700	* Sets the error function to call when an error is flagged on the reader.
	3701	*
	3702	* This should normally be used with mpack_reader_set_context() to register
	3703	* a custom pointer to pass to the error function.
	3704	*
	3705	* See the definition of mpack_reader_error_t for more information about
	3706	* what you can do from an error callback.
	3707	*
	3708	* @see mpack_reader_error_t
	3709	* @param reader The MPack reader.
	3710	* @param error_fn The function to call when an error is flagged on the reader.
	3711	*/
	3712	MPACK_INLINE void mpack_reader_set_error_handler(mpack_reader_t* reader, mpack_reader_error_t error_fn) {
	3713	reader->error_fn = error_fn;
	3714	}
	3715
	3716	/**
	3717	* Sets the teardown function to call when the reader is destroyed.
	3718	*
	3719	* This should normally be used with mpack_reader_set_context() to register
	3720	* a custom pointer to pass to the teardown function.
	3721	*
	3722	* @param reader The MPack reader.
	3723	* @param teardown The function to call when the reader is destroyed.
	3724	*/
	3725	MPACK_INLINE void mpack_reader_set_teardown(mpack_reader_t* reader, mpack_reader_teardown_t teardown) {
	3726	reader->teardown = teardown;
	3727	}
	3728
	3729	/**
	3730	* @}
	3731	*/
	3732
	3733	/**
	3734	* @name Core Reader Functions
	3735	* @{
	3736	*/
	3737
	3738	/**
	3739	* Queries the error state of the MPack reader.
	3740	*
	3741	* If a reader is in an error state, you should discard all data since the
	3742	* last time the error flag was checked. The error flag cannot be cleared.
	3743	*/
	3744	MPACK_INLINE mpack_error_t mpack_reader_error(mpack_reader_t* reader) {
	3745	return reader->error;
	3746	}
	3747
	3748	/**
	3749	* Places the reader in the given error state, calling the error callback if one
	3750	* is set.
	3751	*
	3752	* This allows you to externally flag errors, for example if you are validating
	3753	* data as you read it.
	3754	*
	3755	* If the reader is already in an error state, this call is ignored and no
	3756	* error callback is called.
	3757	*/
	3758	void mpack_reader_flag_error(mpack_reader_t* reader, mpack_error_t error);
	3759
	3760	/**
	3761	* Places the reader in the given error state if the given error is not mpack_ok,
	3762	* returning the resulting error state of the reader.
	3763	*
	3764	* This allows you to externally flag errors, for example if you are validating
	3765	* data as you read it.
	3766	*
	3767	* If the given error is mpack_ok or if the reader is already in an error state,
	3768	* this call is ignored and the actual error state of the reader is returned.
	3769	*/
	3770	MPACK_INLINE mpack_error_t mpack_reader_flag_if_error(mpack_reader_t* reader, mpack_error_t error) {
	3771	if (error != mpack_ok)
	3772	mpack_reader_flag_error(reader, error);
	3773	return mpack_reader_error(reader);
	3774	}
	3775
	3776	/**
	3777	* Returns bytes left in the reader's buffer.
	3778	*
	3779	* If you are done reading MessagePack data but there is other interesting data
	3780	* following it, the reader may have buffered too much data. The number of bytes
	3781	* remaining in the buffer and a pointer to the position of those bytes can be
	3782	* queried here.
	3783	*
	3784	* If you know the length of the MPack chunk beforehand, it's better to instead
	3785	* have your fill function limit the data it reads so that the reader does not
	3786	* have extra data. In this case you can simply check that this returns zero.
	3787	*
	3788	* Returns 0 if the reader is in an error state.
	3789	*
	3790	* @param reader The MPack reader from which to query remaining data.
	3791	* @param data [out] A pointer to the remaining data, or NULL.
	3792	* @return The number of bytes remaining in the buffer.
	3793	*/
	3794	size_t mpack_reader_remaining(mpack_reader_t* reader, const char** data);
	3795
	3796	/**
	3797	* Reads a MessagePack object header (an MPack tag.)
	3798	*
	3799	* If an error occurs, the reader is placed in an error state and a
	3800	* nil tag is returned. If the reader is already in an error state,
	3801	* a nil tag is returned.
	3802	*
	3803	* If the type is compound (i.e. is a map, array, string, binary or
	3804	* extension type), additional reads are required to get the contained
	3805	* data, and the corresponding done function must be called when done.
	3806	*
	3807	* @note Maps in JSON are unordered, so it is recommended not to expect
	3808	* a specific ordering for your map values in case your data is converted
	3809	* to/from JSON.
	3810	*
	3811	* @see mpack_read_bytes()
	3812	* @see mpack_done_array()
	3813	* @see mpack_done_map()
	3814	* @see mpack_done_str()
	3815	* @see mpack_done_bin()
	3816	* @see mpack_done_ext()
	3817	*/
	3818	mpack_tag_t mpack_read_tag(mpack_reader_t* reader);
	3819
	3820	/**
	3821	* Parses the next MessagePack object header (an MPack tag) without
	3822	* advancing the reader.
	3823	*
	3824	* If an error occurs, the reader is placed in an error state and a
	3825	* nil tag is returned. If the reader is already in an error state,
	3826	* a nil tag is returned.
	3827	*
	3828	* @note Maps in JSON are unordered, so it is recommended not to expect
	3829	* a specific ordering for your map values in case your data is converted
	3830	* to/from JSON.
	3831	*
	3832	* @see mpack_read_tag()
	3833	* @see mpack_discard()
	3834	*/
	3835	mpack_tag_t mpack_peek_tag(mpack_reader_t* reader);
	3836
	3837	/**
	3838	* @}
	3839	*/
	3840
	3841	/**
	3842	* @name String and Data Functions
	3843	* @{
	3844	*/
	3845
	3846	/**
	3847	* Skips bytes from the underlying stream. This is used only to
	3848	* skip the contents of a string, binary blob or extension object.
	3849	*/
	3850	void mpack_skip_bytes(mpack_reader_t* reader, size_t count);
	3851
	3852	/**
	3853	* Reads bytes from a string, binary blob or extension object, copying
	3854	* them into the given buffer.
	3855	*
	3856	* A str, bin or ext must have been opened by a call to mpack_read_tag()
	3857	* which yielded one of these types, or by a call to an expect function
	3858	* such as mpack_expect_str() or mpack_expect_bin().
	3859	*
	3860	* If an error occurs, the buffer contents are undefined.
	3861	*
	3862	* This can be called multiple times for a single str, bin or ext
	3863	* to read the data in chunks. The total data read must add up
	3864	* to the size of the object.
	3865	*
	3866	* @param reader The MPack reader
	3867	* @param p The buffer in which to copy the bytes
	3868	* @param count The number of bytes to read
	3869	*/
	3870	void mpack_read_bytes(mpack_reader_t* reader, char* p, size_t count);
	3871
	3872	/**
	3873	* Reads bytes from a string, ensures that the string is valid UTF-8,
	3874	* and copies the bytes into the given buffer.
	3875	*
	3876	* A string must have been opened by a call to mpack_read_tag() which
	3877	* yielded a string, or by a call to an expect function such as
	3878	* mpack_expect_str().
	3879	*
	3880	* The given byte count must match the complete size of the string as
	3881	* returned by the tag or expect function. You must ensure that the
	3882	* buffer fits the data.
	3883	*
	3884	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	3885	* WTF-8. Only pure UTF-8 is allowed.
	3886	*
	3887	* If an error occurs, the buffer contents are undefined.
	3888	*
	3889	* Unlike mpack_read_bytes(), this cannot be used to read the data in
	3890	* chunks (since this might split a character's UTF-8 bytes, and the
	3891	* reader does not keep track of the UTF-8 decoding state between reads.)
	3892	*
	3893	* @throws mpack_error_type if the string contains invalid UTF-8.
	3894	*/
	3895	void mpack_read_utf8(mpack_reader_t* reader, char* p, size_t byte_count);
	3896
	3897	/**
	3898	* Reads bytes from a string, ensures that the string contains no NUL
	3899	* bytes, copies the bytes into the given buffer and adds a null-terminator.
	3900	*
	3901	* A string must have been opened by a call to mpack_read_tag() which
	3902	* yielded a string, or by a call to an expect function such as
	3903	* mpack_expect_str().
	3904	*
	3905	* The given byte count must match the size of the string as returned
	3906	* by the tag or expect function. The string will only be copied if
	3907	* the buffer is large enough to store it.
	3908	*
	3909	* If an error occurs, the buffer will contain an empty string.
	3910	*
	3911	* @note If you know the object will be a string before reading it,
	3912	* it is highly recommended to use mpack_expect_cstr() instead.
	3913	* Alternatively you could use mpack_peek_tag() and call
	3914	* mpack_expect_cstr() if it's a string.
	3915	*
	3916	* @throws mpack_error_too_big if the string plus null-terminator is larger than the given buffer size
	3917	* @throws mpack_error_type if the string contains a null byte.
	3918	*
	3919	* @see mpack_peek_tag()
	3920	* @see mpack_expect_cstr()
	3921	* @see mpack_expect_utf8_cstr()
	3922	*/
	3923	void mpack_read_cstr(mpack_reader_t* reader, char* buf, size_t buffer_size, size_t byte_count);
	3924
	3925	/**
	3926	* Reads bytes from a string, ensures that the string is valid UTF-8
	3927	* with no NUL bytes, copies the bytes into the given buffer and adds a
	3928	* null-terminator.
	3929	*
	3930	* A string must have been opened by a call to mpack_read_tag() which
	3931	* yielded a string, or by a call to an expect function such as
	3932	* mpack_expect_str().
	3933	*
	3934	* The given byte count must match the size of the string as returned
	3935	* by the tag or expect function. The string will only be copied if
	3936	* the buffer is large enough to store it.
	3937	*
	3938	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	3939	* WTF-8. Only pure UTF-8 is allowed, but without the NUL character, since
	3940	* it cannot be represented in a null-terminated string.
	3941	*
	3942	* If an error occurs, the buffer will contain an empty string.
	3943	*
	3944	* @note If you know the object will be a string before reading it,
	3945	* it is highly recommended to use mpack_expect_utf8_cstr() instead.
	3946	* Alternatively you could use mpack_peek_tag() and call
	3947	* mpack_expect_utf8_cstr() if it's a string.
	3948	*
	3949	* @throws mpack_error_too_big if the string plus null-terminator is larger than the given buffer size
	3950	* @throws mpack_error_type if the string contains invalid UTF-8 or a null byte.
	3951	*
	3952	* @see mpack_peek_tag()
	3953	* @see mpack_expect_utf8_cstr()
	3954	*/
	3955	void mpack_read_utf8_cstr(mpack_reader_t* reader, char* buf, size_t buffer_size, size_t byte_count);
	3956
	3957	#ifdef MPACK_MALLOC
	3958	/** @cond */
	3959	// This can optionally add a null-terminator, but it does not check
	3960	// whether the data contains null bytes. This must be done separately
	3961	// in a cstring read function (possibly as part of a UTF-8 check.)
	3962	char* mpack_read_bytes_alloc_impl(mpack_reader_t* reader, size_t count, bool null_terminated);
	3963	/** @endcond */
	3964
	3965	/**
	3966	* Reads bytes from a string, binary blob or extension object, allocating
	3967	* storage for them and returning the allocated pointer.
	3968	*
	3969	* The allocated string must be freed with MPACK_FREE() (or simply free()
	3970	* if MPack's allocator hasn't been customized.)
	3971	*
	3972	* Returns NULL if any error occurs, or if count is zero.
	3973	*/
	3974	MPACK_INLINE char* mpack_read_bytes_alloc(mpack_reader_t* reader, size_t count) {
	3975	return mpack_read_bytes_alloc_impl(reader, count, false);
	3976	}
	3977	#endif
	3978
	3979	/**
	3980	* Reads bytes from a string, binary blob or extension object in-place in
	3981	* the buffer. This can be used to avoid copying the data.
	3982	*
	3983	* A str, bin or ext must have been opened by a call to mpack_read_tag()
	3984	* which yielded one of these types, or by a call to an expect function
	3985	* such as mpack_expect_str() or mpack_expect_bin().
	3986	*
	3987	* If the bytes are from a string, the string is not null-terminated! Use
	3988	* mpack_read_cstr() to copy the string into a buffer and add a null-terminator.
	3989	*
	3990	* The returned pointer is invalidated on the next read, or when the buffer
	3991	* is destroyed.
	3992	*
	3993	* The reader will move data around in the buffer if needed to ensure that
	3994	* the pointer can always be returned, so this should only be used if
	3995	* count is very small compared to the buffer size. If you need to check
	3996	* whether a small size is reasonable (for example you intend to handle small and
	3997	* large sizes differently), you can call mpack_should_read_bytes_inplace().
	3998	*
	3999	* This can be called multiple times for a single str, bin or ext
	4000	* to read the data in chunks. The total data read must add up
	4001	* to the size of the object.
	4002	*
	4003	* NULL is returned if the reader is in an error state.
	4004	*
	4005	* @throws mpack_error_too_big if the requested size is larger than the buffer size
	4006	*
	4007	* @see mpack_should_read_bytes_inplace()
	4008	*/
	4009	const char* mpack_read_bytes_inplace(mpack_reader_t* reader, size_t count);
	4010
	4011	/**
	4012	* Reads bytes from a string in-place in the buffer and ensures they are
	4013	* valid UTF-8. This can be used to avoid copying the data.
	4014	*
	4015	* A string must have been opened by a call to mpack_read_tag() which
	4016	* yielded a string, or by a call to an expect function such as
	4017	* mpack_expect_str().
	4018	*
	4019	* The string is not null-terminated! Use mpack_read_utf8_cstr() to
	4020	* copy the string into a buffer and add a null-terminator.
	4021	*
	4022	* The returned pointer is invalidated on the next read, or when the buffer
	4023	* is destroyed.
	4024	*
	4025	* The reader will move data around in the buffer if needed to ensure that
	4026	* the pointer can always be returned, so this should only be used if
	4027	* count is very small compared to the buffer size. If you need to check
	4028	* whether a small size is reasonable (for example you intend to handle small and
	4029	* large sizes differently), you can call mpack_should_read_bytes_inplace().
	4030	*
	4031	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	4032	* WTF-8. Only pure UTF-8 is allowed.
	4033	*
	4034	* Unlike mpack_read_bytes_inplace(), this cannot be used to read the data in
	4035	* chunks (since this might split a character's UTF-8 bytes, and the
	4036	* reader does not keep track of the UTF-8 decoding state between reads.)
	4037	*
	4038	* NULL is returned if the reader is in an error state.
	4039	*
	4040	* @throws mpack_error_type if the string contains invalid UTF-8
	4041	* @throws mpack_error_too_big if the requested size is larger than the buffer size
	4042	*
	4043	* @see mpack_should_read_bytes_inplace()
	4044	*/
	4045	const char* mpack_read_utf8_inplace(mpack_reader_t* reader, size_t count);
	4046
	4047	/**
	4048	* Returns true if it's a good idea to read the given number of bytes
	4049	* in-place.
	4050	*
	4051	* If the read will be larger than some small fraction of the buffer size,
	4052	* this will return false to avoid shuffling too much data back and forth
	4053	* in the buffer.
	4054	*
	4055	* Use this if you're expecting arbitrary size data, and you want to read
	4056	* in-place for the best performance when possible but will fall back to
	4057	* a normal read if the data is too large.
	4058	*
	4059	* @see mpack_read_bytes_inplace()
	4060	*/
	4061	MPACK_INLINE bool mpack_should_read_bytes_inplace(mpack_reader_t* reader, size_t count) {
	4062	return (reader->size == 0 \|\| count <= reader->size / MPACK_READER_SMALL_FRACTION_DENOMINATOR);
	4063	}
	4064
	4065	#if MPACK_EXTENSIONS
	4066	/**
	4067	* Reads a timestamp contained in an ext object of the given size, closing the
	4068	* ext type.
	4069	*
	4070	* An ext object of exttype @ref MPACK_EXTTYPE_TIMESTAMP must have been opened
	4071	* by a call to e.g. mpack_read_tag() or mpack_expect_ext().
	4072	*
	4073	* You must NOT call mpack_done_ext() after calling this. A timestamp ext
	4074	* object can only contain a single timestamp value, so this calls
	4075	* mpack_done_ext() automatically.
	4076	*
	4077	* @note This requires @ref MPACK_EXTENSIONS.
	4078	*
	4079	* @throws mpack_error_invalid if the size is not one of the supported
	4080	* timestamp sizes, or if the nanoseconds are out of range.
	4081	*/
	4082	mpack_timestamp_t mpack_read_timestamp(mpack_reader_t* reader, size_t size);
	4083	#endif
	4084
	4085	/**
	4086	* @}
	4087	*/
	4088
	4089	/**
	4090	* @name Core Reader Functions
	4091	* @{
	4092	*/
	4093
	4094	#if MPACK_READ_TRACKING
	4095	/**
	4096	* Finishes reading the given type.
	4097	*
	4098	* This will track reads to ensure that the correct number of elements
	4099	* or bytes are read.
	4100	*/
	4101	void mpack_done_type(mpack_reader_t* reader, mpack_type_t type);
	4102	#else
	4103	MPACK_INLINE void mpack_done_type(mpack_reader_t* reader, mpack_type_t type) {
	4104	MPACK_UNUSED(reader);
	4105	MPACK_UNUSED(type);
	4106	}
	4107	#endif
	4108
	4109	/**
	4110	* Finishes reading an array.
	4111	*
	4112	* This will track reads to ensure that the correct number of elements are read.
	4113	*/
	4114	MPACK_INLINE void mpack_done_array(mpack_reader_t* reader) {
	4115	mpack_done_type(reader, mpack_type_array);
	4116	}
	4117
	4118	/**
	4119	* @fn mpack_done_map(mpack_reader_t* reader)
	4120	*
	4121	* Finishes reading a map.
	4122	*
	4123	* This will track reads to ensure that the correct number of elements are read.
	4124	*/
	4125	MPACK_INLINE void mpack_done_map(mpack_reader_t* reader) {
	4126	mpack_done_type(reader, mpack_type_map);
	4127	}
	4128
	4129	/**
	4130	* @fn mpack_done_str(mpack_reader_t* reader)
	4131	*
	4132	* Finishes reading a string.
	4133	*
	4134	* This will track reads to ensure that the correct number of bytes are read.
	4135	*/
	4136	MPACK_INLINE void mpack_done_str(mpack_reader_t* reader) {
	4137	mpack_done_type(reader, mpack_type_str);
	4138	}
	4139
	4140	/**
	4141	* @fn mpack_done_bin(mpack_reader_t* reader)
	4142	*
	4143	* Finishes reading a binary data blob.
	4144	*
	4145	* This will track reads to ensure that the correct number of bytes are read.
	4146	*/
	4147	MPACK_INLINE void mpack_done_bin(mpack_reader_t* reader) {
	4148	mpack_done_type(reader, mpack_type_bin);
	4149	}
	4150
	4151	#if MPACK_EXTENSIONS
	4152	/**
	4153	* @fn mpack_done_ext(mpack_reader_t* reader)
	4154	*
	4155	* Finishes reading an extended type binary data blob.
	4156	*
	4157	* This will track reads to ensure that the correct number of bytes are read.
	4158	*
	4159	* @note This requires @ref MPACK_EXTENSIONS.
	4160	*/
	4161	MPACK_INLINE void mpack_done_ext(mpack_reader_t* reader) {
	4162	mpack_done_type(reader, mpack_type_ext);
	4163	}
	4164	#endif
	4165
	4166	/**
	4167	* Reads and discards the next object. This will read and discard all
	4168	* contained data as well if it is a compound type.
	4169	*/
	4170	void mpack_discard(mpack_reader_t* reader);
	4171
	4172	/**
	4173	* @}
	4174	*/
	4175
	4176	/** @cond */
	4177
	4178	#if MPACK_DEBUG && MPACK_STDIO
	4179	/**
	4180	* @name Debugging Functions
	4181	* @{
	4182	*/
	4183	/*
	4184	* Converts a blob of MessagePack to a pseudo-JSON string for debugging
	4185	* purposes, placing the result in the given buffer with a null-terminator.
	4186	*
	4187	* If the buffer does not have enough space, the result will be truncated (but
	4188	* it is guaranteed to be null-terminated.)
	4189	*
	4190	* This is only available in debug mode, and only if stdio is available (since
	4191	* it uses snprintf().) It's strictly for debugging purposes.
	4192	*/
	4193	void mpack_print_data_to_buffer(const char* data, size_t data_size, char* buffer, size_t buffer_size);
	4194
	4195	/*
	4196	* Converts a node to pseudo-JSON for debugging purposes, calling the given
	4197	* callback as many times as is necessary to output the character data.
	4198	*
	4199	* No null-terminator or trailing newline will be written.
	4200	*
	4201	* This is only available in debug mode, and only if stdio is available (since
	4202	* it uses snprintf().) It's strictly for debugging purposes.
	4203	*/
	4204	void mpack_print_data_to_callback(const char* data, size_t size, mpack_print_callback_t callback, void* context);
	4205
	4206	/*
	4207	* Converts a blob of MessagePack to pseudo-JSON for debugging purposes
	4208	* and pretty-prints it to the given file.
	4209	*/
	4210	void mpack_print_data_to_file(const char* data, size_t len, FILE* file);
	4211
	4212	/*
	4213	* Converts a blob of MessagePack to pseudo-JSON for debugging purposes
	4214	* and pretty-prints it to stdout.
	4215	*/
	4216	MPACK_INLINE void mpack_print_data_to_stdout(const char* data, size_t len) {
	4217	mpack_print_data_to_file(data, len, stdout);
	4218	}
	4219
	4220	/*
	4221	* Converts the MessagePack contained in the given `FILE*` to pseudo-JSON for
	4222	* debugging purposes, calling the given callback as many times as is necessary
	4223	* to output the character data.
	4224	*/
	4225	void mpack_print_stdfile_to_callback(FILE* file, mpack_print_callback_t callback, void* context);
	4226
	4227	/*
	4228	* Deprecated.
	4229	*
	4230	* \deprecated Renamed to mpack_print_data_to_stdout().
	4231	*/
	4232	MPACK_INLINE void mpack_print(const char* data, size_t len) {
	4233	mpack_print_data_to_stdout(data, len);
	4234	}
	4235
	4236	/**
	4237	* @}
	4238	*/
	4239	#endif
	4240
	4241	/** @endcond */
	4242
	4243	/**
	4244	* @}
	4245	*/
	4246
	4247
	4248
	4249	#if MPACK_INTERNAL
	4250
	4251	bool mpack_reader_ensure_straddle(mpack_reader_t* reader, size_t count);
	4252
	4253	/*
	4254	* Ensures there are at least @c count bytes left in the
	4255	* data, raising an error and returning false if more
	4256	* data cannot be made available.
	4257	*/
	4258	MPACK_INLINE bool mpack_reader_ensure(mpack_reader_t* reader, size_t count) {
	4259	mpack_assert(count != 0, "cannot ensure zero bytes!");
	4260	mpack_assert(reader->error == mpack_ok, "reader cannot be in an error state!");
	4261
	4262	if (count <= (size_t)(reader->end - reader->data))
	4263	return true;
	4264	return mpack_reader_ensure_straddle(reader, count);
	4265	}
	4266
	4267	void mpack_read_native_straddle(mpack_reader_t* reader, char* p, size_t count);
	4268
	4269	// Reads count bytes into p, deferring to mpack_read_native_straddle() if more
	4270	// bytes are needed than are available in the buffer.
	4271	MPACK_INLINE void mpack_read_native(mpack_reader_t* reader, char* p, size_t count) {
	4272	mpack_assert(count == 0 \|\| p != NULL, "data pointer for %i bytes is NULL", (int)count);
	4273
	4274	if (count > (size_t)(reader->end - reader->data)) {
	4275	mpack_read_native_straddle(reader, p, count);
	4276	} else {
	4277	mpack_memcpy(p, reader->data, count);
	4278	reader->data += count;
	4279	}
	4280	}
	4281
	4282	#if MPACK_READ_TRACKING
	4283	#define MPACK_READER_TRACK(reader, error_expr) \
	4284	(((reader)->error == mpack_ok) ? mpack_reader_flag_if_error((reader), (error_expr)) : (reader)->error)
	4285	#else
	4286	#define MPACK_READER_TRACK(reader, error_expr) (MPACK_UNUSED(reader), mpack_ok)
	4287	#endif
	4288
	4289	MPACK_INLINE mpack_error_t mpack_reader_track_element(mpack_reader_t* reader) {
	4290	return MPACK_READER_TRACK(reader, mpack_track_element(&reader->track, true));
	4291	}
	4292
	4293	MPACK_INLINE mpack_error_t mpack_reader_track_peek_element(mpack_reader_t* reader) {
	4294	return MPACK_READER_TRACK(reader, mpack_track_peek_element(&reader->track, true));
	4295	}
	4296
	4297	MPACK_INLINE mpack_error_t mpack_reader_track_bytes(mpack_reader_t* reader, uint64_t count) {
	4298	MPACK_UNUSED(count);
	4299	return MPACK_READER_TRACK(reader, mpack_track_bytes(&reader->track, true, count));
	4300	}
	4301
	4302	MPACK_INLINE mpack_error_t mpack_reader_track_str_bytes_all(mpack_reader_t* reader, uint64_t count) {
	4303	MPACK_UNUSED(count);
	4304	return MPACK_READER_TRACK(reader, mpack_track_str_bytes_all(&reader->track, true, count));
	4305	}
	4306
	4307	#endif
	4308
	4309
	4310
	4311	#endif
	4312
	4313	MPACK_HEADER_END
	4314
	4315	#endif
	4316
	4317
	4318	/* mpack/mpack-expect.h.h */
	4319
	4320	/**
	4321	* @file
	4322	*
	4323	* Declares the MPack static Expect API.
	4324	*/
	4325
	4326	#ifndef MPACK_EXPECT_H
	4327	#define MPACK_EXPECT_H 1
	4328
	4329	/* #include "mpack-reader.h" */
	4330
	4331	MPACK_HEADER_START
	4332
	4333	#if MPACK_EXPECT
	4334
	4335	#if !MPACK_READER
	4336	#error "MPACK_EXPECT requires MPACK_READER."
	4337	#endif
	4338
	4339	/**
	4340	* @defgroup expect Expect API
	4341	*
	4342	* The MPack Expect API allows you to easily read MessagePack data when you
	4343	* expect it to follow a predefined schema.
	4344	*
	4345	* @note If you are not writing code for an embedded device (or otherwise do
	4346	* not need maximum performance with minimal memory usage), you should not use
	4347	* this. You probably want to use the @link node Node API@endlink instead.
	4348	*
	4349	* See @ref docs/expect.md for examples.
	4350	*
	4351	* The main purpose of the Expect API is convenience, so the API is lax. It
	4352	* automatically converts between similar types where there is no loss of
	4353	* precision.
	4354	*
	4355	* When using any of the expect functions, if the type or value of what was
	4356	* read does not match what is expected, @ref mpack_error_type is raised.
	4357	*
	4358	* @{
	4359	*/
	4360
	4361	/**
	4362	* @name Basic Number Functions
	4363	* @{
	4364	*/
	4365
	4366	/**
	4367	* Reads an 8-bit unsigned integer.
	4368	*
	4369	* The underlying type may be an integer type of any size and signedness,
	4370	* as long as the value can be represented in an 8-bit unsigned int.
	4371	*
	4372	* Returns zero if an error occurs.
	4373	*/
	4374	uint8_t mpack_expect_u8(mpack_reader_t* reader);
	4375
	4376	/**
	4377	* Reads a 16-bit unsigned integer.
	4378	*
	4379	* The underlying type may be an integer type of any size and signedness,
	4380	* as long as the value can be represented in a 16-bit unsigned int.
	4381	*
	4382	* Returns zero if an error occurs.
	4383	*/
	4384	uint16_t mpack_expect_u16(mpack_reader_t* reader);
	4385
	4386	/**
	4387	* Reads a 32-bit unsigned integer.
	4388	*
	4389	* The underlying type may be an integer type of any size and signedness,
	4390	* as long as the value can be represented in a 32-bit unsigned int.
	4391	*
	4392	* Returns zero if an error occurs.
	4393	*/
	4394	uint32_t mpack_expect_u32(mpack_reader_t* reader);
	4395
	4396	/**
	4397	* Reads a 64-bit unsigned integer.
	4398	*
	4399	* The underlying type may be an integer type of any size and signedness,
	4400	* as long as the value can be represented in a 64-bit unsigned int.
	4401	*
	4402	* Returns zero if an error occurs.
	4403	*/
	4404	uint64_t mpack_expect_u64(mpack_reader_t* reader);
	4405
	4406	/**
	4407	* Reads an 8-bit signed integer.
	4408	*
	4409	* The underlying type may be an integer type of any size and signedness,
	4410	* as long as the value can be represented in an 8-bit signed int.
	4411	*
	4412	* Returns zero if an error occurs.
	4413	*/
	4414	int8_t mpack_expect_i8(mpack_reader_t* reader);
	4415
	4416	/**
	4417	* Reads a 16-bit signed integer.
	4418	*
	4419	* The underlying type may be an integer type of any size and signedness,
	4420	* as long as the value can be represented in a 16-bit signed int.
	4421	*
	4422	* Returns zero if an error occurs.
	4423	*/
	4424	int16_t mpack_expect_i16(mpack_reader_t* reader);
	4425
	4426	/**
	4427	* Reads a 32-bit signed integer.
	4428	*
	4429	* The underlying type may be an integer type of any size and signedness,
	4430	* as long as the value can be represented in a 32-bit signed int.
	4431	*
	4432	* Returns zero if an error occurs.
	4433	*/
	4434	int32_t mpack_expect_i32(mpack_reader_t* reader);
	4435
	4436	/**
	4437	* Reads a 64-bit signed integer.
	4438	*
	4439	* The underlying type may be an integer type of any size and signedness,
	4440	* as long as the value can be represented in a 64-bit signed int.
	4441	*
	4442	* Returns zero if an error occurs.
	4443	*/
	4444	int64_t mpack_expect_i64(mpack_reader_t* reader);
	4445
	4446	/**
	4447	* Reads a number, returning the value as a float. The underlying value can be an
	4448	* integer, float or double; the value is converted to a float.
	4449	*
	4450	* @note Reading a double or a large integer with this function can incur a
	4451	* loss of precision.
	4452	*
	4453	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	4454	*/
	4455	float mpack_expect_float(mpack_reader_t* reader);
	4456
	4457	/**
	4458	* Reads a number, returning the value as a double. The underlying value can be an
	4459	* integer, float or double; the value is converted to a double.
	4460	*
	4461	* @note Reading a very large integer with this function can incur a
	4462	* loss of precision.
	4463	*
	4464	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	4465	*/
	4466	double mpack_expect_double(mpack_reader_t* reader);
	4467
	4468	/**
	4469	* Reads a float. The underlying value must be a float, not a double or an integer.
	4470	* This ensures no loss of precision can occur.
	4471	*
	4472	* @throws mpack_error_type if the underlying value is not a float.
	4473	*/
	4474	float mpack_expect_float_strict(mpack_reader_t* reader);
	4475
	4476	/**
	4477	* Reads a double. The underlying value must be a float or double, not an integer.
	4478	* This ensures no loss of precision can occur.
	4479	*
	4480	* @throws mpack_error_type if the underlying value is not a float or double.
	4481	*/
	4482	double mpack_expect_double_strict(mpack_reader_t* reader);
	4483
	4484	/**
	4485	* @}
	4486	*/
	4487
	4488	/**
	4489	* @name Ranged Number Functions
	4490	* @{
	4491	*/
	4492
	4493	/**
	4494	* Reads an 8-bit unsigned integer, ensuring that it falls within the given range.
	4495	*
	4496	* The underlying type may be an integer type of any size and signedness,
	4497	* as long as the value can be represented in an 8-bit unsigned int.
	4498	*
	4499	* Returns min_value if an error occurs.
	4500	*/
	4501	uint8_t mpack_expect_u8_range(mpack_reader_t* reader, uint8_t min_value, uint8_t max_value);
	4502
	4503	/**
	4504	* Reads a 16-bit unsigned integer, ensuring that it falls within the given range.
	4505	*
	4506	* The underlying type may be an integer type of any size and signedness,
	4507	* as long as the value can be represented in a 16-bit unsigned int.
	4508	*
	4509	* Returns min_value if an error occurs.
	4510	*/
	4511	uint16_t mpack_expect_u16_range(mpack_reader_t* reader, uint16_t min_value, uint16_t max_value);
	4512
	4513	/**
	4514	* Reads a 32-bit unsigned integer, ensuring that it falls within the given range.
	4515	*
	4516	* The underlying type may be an integer type of any size and signedness,
	4517	* as long as the value can be represented in a 32-bit unsigned int.
	4518	*
	4519	* Returns min_value if an error occurs.
	4520	*/
	4521	uint32_t mpack_expect_u32_range(mpack_reader_t* reader, uint32_t min_value, uint32_t max_value);
	4522
	4523	/**
	4524	* Reads a 64-bit unsigned integer, ensuring that it falls within the given range.
	4525	*
	4526	* The underlying type may be an integer type of any size and signedness,
	4527	* as long as the value can be represented in a 64-bit unsigned int.
	4528	*
	4529	* Returns min_value if an error occurs.
	4530	*/
	4531	uint64_t mpack_expect_u64_range(mpack_reader_t* reader, uint64_t min_value, uint64_t max_value);
	4532
	4533	/**
	4534	* Reads an unsigned integer, ensuring that it falls within the given range.
	4535	*
	4536	* The underlying type may be an integer type of any size and signedness,
	4537	* as long as the value can be represented in an unsigned int.
	4538	*
	4539	* Returns min_value if an error occurs.
	4540	*/
	4541	MPACK_INLINE unsigned int mpack_expect_uint_range(mpack_reader_t* reader, unsigned int min_value, unsigned int max_value) {
	4542	// This should be true at compile-time, so this just wraps the 32-bit
	4543	// function. We fallback to 64-bit if for some reason sizeof(int) isn't 4.
	4544	if (sizeof(unsigned int) == 4)
	4545	return (unsigned int)mpack_expect_u32_range(reader, (uint32_t)min_value, (uint32_t)max_value);
	4546	return (unsigned int)mpack_expect_u64_range(reader, min_value, max_value);
	4547	}
	4548
	4549	/**
	4550	* Reads an 8-bit unsigned integer, ensuring that it is at most @a max_value.
	4551	*
	4552	* The underlying type may be an integer type of any size and signedness,
	4553	* as long as the value can be represented in an 8-bit unsigned int.
	4554	*
	4555	* Returns 0 if an error occurs.
	4556	*/
	4557	MPACK_INLINE uint8_t mpack_expect_u8_max(mpack_reader_t* reader, uint8_t max_value) {
	4558	return mpack_expect_u8_range(reader, 0, max_value);
	4559	}
	4560
	4561	/**
	4562	* Reads a 16-bit unsigned integer, ensuring that it is at most @a max_value.
	4563	*
	4564	* The underlying type may be an integer type of any size and signedness,
	4565	* as long as the value can be represented in a 16-bit unsigned int.
	4566	*
	4567	* Returns 0 if an error occurs.
	4568	*/
	4569	MPACK_INLINE uint16_t mpack_expect_u16_max(mpack_reader_t* reader, uint16_t max_value) {
	4570	return mpack_expect_u16_range(reader, 0, max_value);
	4571	}
	4572
	4573	/**
	4574	* Reads a 32-bit unsigned integer, ensuring that it is at most @a max_value.
	4575	*
	4576	* The underlying type may be an integer type of any size and signedness,
	4577	* as long as the value can be represented in a 32-bit unsigned int.
	4578	*
	4579	* Returns 0 if an error occurs.
	4580	*/
	4581	MPACK_INLINE uint32_t mpack_expect_u32_max(mpack_reader_t* reader, uint32_t max_value) {
	4582	return mpack_expect_u32_range(reader, 0, max_value);
	4583	}
	4584
	4585	/**
	4586	* Reads a 64-bit unsigned integer, ensuring that it is at most @a max_value.
	4587	*
	4588	* The underlying type may be an integer type of any size and signedness,
	4589	* as long as the value can be represented in a 64-bit unsigned int.
	4590	*
	4591	* Returns 0 if an error occurs.
	4592	*/
	4593	MPACK_INLINE uint64_t mpack_expect_u64_max(mpack_reader_t* reader, uint64_t max_value) {
	4594	return mpack_expect_u64_range(reader, 0, max_value);
	4595	}
	4596
	4597	/**
	4598	* Reads an unsigned integer, ensuring that it is at most @a max_value.
	4599	*
	4600	* The underlying type may be an integer type of any size and signedness,
	4601	* as long as the value can be represented in an unsigned int.
	4602	*
	4603	* Returns 0 if an error occurs.
	4604	*/
	4605	MPACK_INLINE unsigned int mpack_expect_uint_max(mpack_reader_t* reader, unsigned int max_value) {
	4606	return mpack_expect_uint_range(reader, 0, max_value);
	4607	}
	4608
	4609	/**
	4610	* Reads an 8-bit signed integer, ensuring that it falls within the given range.
	4611	*
	4612	* The underlying type may be an integer type of any size and signedness,
	4613	* as long as the value can be represented in an 8-bit signed int.
	4614	*
	4615	* Returns min_value if an error occurs.
	4616	*/
	4617	int8_t mpack_expect_i8_range(mpack_reader_t* reader, int8_t min_value, int8_t max_value);
	4618
	4619	/**
	4620	* Reads a 16-bit signed integer, ensuring that it falls within the given range.
	4621	*
	4622	* The underlying type may be an integer type of any size and signedness,
	4623	* as long as the value can be represented in a 16-bit signed int.
	4624	*
	4625	* Returns min_value if an error occurs.
	4626	*/
	4627	int16_t mpack_expect_i16_range(mpack_reader_t* reader, int16_t min_value, int16_t max_value);
	4628
	4629	/**
	4630	* Reads a 32-bit signed integer, ensuring that it falls within the given range.
	4631	*
	4632	* The underlying type may be an integer type of any size and signedness,
	4633	* as long as the value can be represented in a 32-bit signed int.
	4634	*
	4635	* Returns min_value if an error occurs.
	4636	*/
	4637	int32_t mpack_expect_i32_range(mpack_reader_t* reader, int32_t min_value, int32_t max_value);
	4638
	4639	/**
	4640	* Reads a 64-bit signed integer, ensuring that it falls within the given range.
	4641	*
	4642	* The underlying type may be an integer type of any size and signedness,
	4643	* as long as the value can be represented in a 64-bit signed int.
	4644	*
	4645	* Returns min_value if an error occurs.
	4646	*/
	4647	int64_t mpack_expect_i64_range(mpack_reader_t* reader, int64_t min_value, int64_t max_value);
	4648
	4649	/**
	4650	* Reads a signed integer, ensuring that it falls within the given range.
	4651	*
	4652	* The underlying type may be an integer type of any size and signedness,
	4653	* as long as the value can be represented in a signed int.
	4654	*
	4655	* Returns min_value if an error occurs.
	4656	*/
	4657	MPACK_INLINE int mpack_expect_int_range(mpack_reader_t* reader, int min_value, int max_value) {
	4658	// This should be true at compile-time, so this just wraps the 32-bit
	4659	// function. We fallback to 64-bit if for some reason sizeof(int) isn't 4.
	4660	if (sizeof(int) == 4)
	4661	return (int)mpack_expect_i32_range(reader, (int32_t)min_value, (int32_t)max_value);
	4662	return (int)mpack_expect_i64_range(reader, min_value, max_value);
	4663	}
	4664
	4665	/**
	4666	* Reads an 8-bit signed integer, ensuring that it is at least zero and at
	4667	* most @a max_value.
	4668	*
	4669	* The underlying type may be an integer type of any size and signedness,
	4670	* as long as the value can be represented in an 8-bit signed int.
	4671	*
	4672	* Returns 0 if an error occurs.
	4673	*/
	4674	MPACK_INLINE int8_t mpack_expect_i8_max(mpack_reader_t* reader, int8_t max_value) {
	4675	return mpack_expect_i8_range(reader, 0, max_value);
	4676	}
	4677
	4678	/**
	4679	* Reads a 16-bit signed integer, ensuring that it is at least zero and at
	4680	* most @a max_value.
	4681	*
	4682	* The underlying type may be an integer type of any size and signedness,
	4683	* as long as the value can be represented in a 16-bit signed int.
	4684	*
	4685	* Returns 0 if an error occurs.
	4686	*/
	4687	MPACK_INLINE int16_t mpack_expect_i16_max(mpack_reader_t* reader, int16_t max_value) {
	4688	return mpack_expect_i16_range(reader, 0, max_value);
	4689	}
	4690
	4691	/**
	4692	* Reads a 32-bit signed integer, ensuring that it is at least zero and at
	4693	* most @a max_value.
	4694	*
	4695	* The underlying type may be an integer type of any size and signedness,
	4696	* as long as the value can be represented in a 32-bit signed int.
	4697	*
	4698	* Returns 0 if an error occurs.
	4699	*/
	4700	MPACK_INLINE int32_t mpack_expect_i32_max(mpack_reader_t* reader, int32_t max_value) {
	4701	return mpack_expect_i32_range(reader, 0, max_value);
	4702	}
	4703
	4704	/**
	4705	* Reads a 64-bit signed integer, ensuring that it is at least zero and at
	4706	* most @a max_value.
	4707	*
	4708	* The underlying type may be an integer type of any size and signedness,
	4709	* as long as the value can be represented in a 64-bit signed int.
	4710	*
	4711	* Returns 0 if an error occurs.
	4712	*/
	4713	MPACK_INLINE int64_t mpack_expect_i64_max(mpack_reader_t* reader, int64_t max_value) {
	4714	return mpack_expect_i64_range(reader, 0, max_value);
	4715	}
	4716
	4717	/**
	4718	* Reads an int, ensuring that it is at least zero and at most @a max_value.
	4719	*
	4720	* The underlying type may be an integer type of any size and signedness,
	4721	* as long as the value can be represented in a signed int.
	4722	*
	4723	* Returns 0 if an error occurs.
	4724	*/
	4725	MPACK_INLINE int mpack_expect_int_max(mpack_reader_t* reader, int max_value) {
	4726	return mpack_expect_int_range(reader, 0, max_value);
	4727	}
	4728
	4729	/**
	4730	* Reads a number, ensuring that it falls within the given range and returning
	4731	* the value as a float. The underlying value can be an integer, float or
	4732	* double; the value is converted to a float.
	4733	*
	4734	* @note Reading a double or a large integer with this function can incur a
	4735	* loss of precision.
	4736	*
	4737	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	4738	*/
	4739	float mpack_expect_float_range(mpack_reader_t* reader, float min_value, float max_value);
	4740
	4741	/**
	4742	* Reads a number, ensuring that it falls within the given range and returning
	4743	* the value as a double. The underlying value can be an integer, float or
	4744	* double; the value is converted to a double.
	4745	*
	4746	* @note Reading a very large integer with this function can incur a
	4747	* loss of precision.
	4748	*
	4749	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	4750	*/
	4751	double mpack_expect_double_range(mpack_reader_t* reader, double min_value, double max_value);
	4752
	4753	/**
	4754	* @}
	4755	*/
	4756
	4757
	4758
	4759	// These are additional Basic Number functions that wrap inline range functions.
	4760
	4761	/**
	4762	* @name Basic Number Functions
	4763	* @{
	4764	*/
	4765
	4766	/**
	4767	* Reads an unsigned int.
	4768	*
	4769	* The underlying type may be an integer type of any size and signedness,
	4770	* as long as the value can be represented in an unsigned int.
	4771	*
	4772	* Returns zero if an error occurs.
	4773	*/
	4774	MPACK_INLINE unsigned int mpack_expect_uint(mpack_reader_t* reader) {
	4775
	4776	// This should be true at compile-time, so this just wraps the 32-bit function.
	4777	if (sizeof(unsigned int) == 4)
	4778	return (unsigned int)mpack_expect_u32(reader);
	4779
	4780	// Otherwise we wrap the max function to ensure it fits.
	4781	return (unsigned int)mpack_expect_u64_max(reader, UINT_MAX);
	4782
	4783	}
	4784
	4785	/**
	4786	* Reads a signed int.
	4787	*
	4788	* The underlying type may be an integer type of any size and signedness,
	4789	* as long as the value can be represented in a signed int.
	4790	*
	4791	* Returns zero if an error occurs.
	4792	*/
	4793	MPACK_INLINE int mpack_expect_int(mpack_reader_t* reader) {
	4794
	4795	// This should be true at compile-time, so this just wraps the 32-bit function.
	4796	if (sizeof(int) == 4)
	4797	return (int)mpack_expect_i32(reader);
	4798
	4799	// Otherwise we wrap the range function to ensure it fits.
	4800	return (int)mpack_expect_i64_range(reader, INT_MIN, INT_MAX);
	4801
	4802	}
	4803
	4804	/**
	4805	* @}
	4806	*/
	4807
	4808
	4809
	4810	/**
	4811	* @name Matching Number Functions
	4812	* @{
	4813	*/
	4814
	4815	/**
	4816	* Reads an unsigned integer, ensuring that it exactly matches the given value.
	4817	*
	4818	* mpack_error_type is raised if the value is not representable as an unsigned
	4819	* integer or if it does not exactly match the given value.
	4820	*/
	4821	void mpack_expect_uint_match(mpack_reader_t* reader, uint64_t value);
	4822
	4823	/**
	4824	* Reads a signed integer, ensuring that it exactly matches the given value.
	4825	*
	4826	* mpack_error_type is raised if the value is not representable as a signed
	4827	* integer or if it does not exactly match the given value.
	4828	*/
	4829	void mpack_expect_int_match(mpack_reader_t* reader, int64_t value);
	4830
	4831	/**
	4832	* @name Other Basic Types
	4833	* @{
	4834	*/
	4835
	4836	/**
	4837	* Reads a nil, raising @ref mpack_error_type if the value is not nil.
	4838	*/
	4839	void mpack_expect_nil(mpack_reader_t* reader);
	4840
	4841	/**
	4842	* Reads a boolean.
	4843	*
	4844	* @note Integers will raise mpack_error_type; the value must be strictly a boolean.
	4845	*/
	4846	bool mpack_expect_bool(mpack_reader_t* reader);
	4847
	4848	/**
	4849	* Reads a boolean, raising @ref mpack_error_type if its value is not @c true.
	4850	*/
	4851	void mpack_expect_true(mpack_reader_t* reader);
	4852
	4853	/**
	4854	* Reads a boolean, raising @ref mpack_error_type if its value is not @c false.
	4855	*/
	4856	void mpack_expect_false(mpack_reader_t* reader);
	4857
	4858	/**
	4859	* @}
	4860	*/
	4861
	4862	/**
	4863	* @name Extension Functions
	4864	* @{
	4865	*/
	4866
	4867	#if MPACK_EXTENSIONS
	4868	/**
	4869	* Reads a timestamp.
	4870	*
	4871	* @note This requires @ref MPACK_EXTENSIONS.
	4872	*/
	4873	mpack_timestamp_t mpack_expect_timestamp(mpack_reader_t* reader);
	4874
	4875	/**
	4876	* Reads a timestamp in seconds, truncating the nanoseconds (if any).
	4877	*
	4878	* @note This requires @ref MPACK_EXTENSIONS.
	4879	*/
	4880	int64_t mpack_expect_timestamp_truncate(mpack_reader_t* reader);
	4881	#endif
	4882
	4883	/**
	4884	* @}
	4885	*/
	4886
	4887	/**
	4888	* @name Compound Types
	4889	* @{
	4890	*/
	4891
	4892	/**
	4893	* Reads the start of a map, returning its element count.
	4894	*
	4895	* A number of values follow equal to twice the element count of the map,
	4896	* alternating between keys and values. @ref mpack_done_map() must be called
	4897	* once all elements have been read.
	4898	*
	4899	* @note Maps in JSON are unordered, so it is recommended not to expect
	4900	* a specific ordering for your map values in case your data is converted
	4901	* to/from JSON.
	4902	*
	4903	* @warning This call is dangerous! It does not have a size limit, and it
	4904	* does not have any way of checking whether there is enough data in the
	4905	* message (since the data could be coming from a stream.) When looping
	4906	* through the map's contents, you must check for errors on each iteration
	4907	* of the loop. Otherwise an attacker could craft a message declaring a map
	4908	* of a billion elements which would throw your parsing code into an
	4909	* infinite loop! You should strongly consider using mpack_expect_map_max()
	4910	* with a safe maximum size instead.
	4911	*
	4912	* @throws mpack_error_type if the value is not a map.
	4913	*/
	4914	uint32_t mpack_expect_map(mpack_reader_t* reader);
	4915
	4916	/**
	4917	* Reads the start of a map with a number of elements in the given range, returning
	4918	* its element count.
	4919	*
	4920	* A number of values follow equal to twice the element count of the map,
	4921	* alternating between keys and values. @ref mpack_done_map() must be called
	4922	* once all elements have been read.
	4923	*
	4924	* @note Maps in JSON are unordered, so it is recommended not to expect
	4925	* a specific ordering for your map values in case your data is converted
	4926	* to/from JSON.
	4927	*
	4928	* min_count is returned if an error occurs.
	4929	*
	4930	* @throws mpack_error_type if the value is not a map or if its size does
	4931	* not fall within the given range.
	4932	*/
	4933	uint32_t mpack_expect_map_range(mpack_reader_t* reader, uint32_t min_count, uint32_t max_count);
	4934
	4935	/**
	4936	* Reads the start of a map with a number of elements at most @a max_count,
	4937	* returning its element count.
	4938	*
	4939	* A number of values follow equal to twice the element count of the map,
	4940	* alternating between keys and values. @ref mpack_done_map() must be called
	4941	* once all elements have been read.
	4942	*
	4943	* @note Maps in JSON are unordered, so it is recommended not to expect
	4944	* a specific ordering for your map values in case your data is converted
	4945	* to/from JSON.
	4946	*
	4947	* Zero is returned if an error occurs.
	4948	*
	4949	* @throws mpack_error_type if the value is not a map or if its size is
	4950	* greater than max_count.
	4951	*/
	4952	MPACK_INLINE uint32_t mpack_expect_map_max(mpack_reader_t* reader, uint32_t max_count) {
	4953	return mpack_expect_map_range(reader, 0, max_count);
	4954	}
	4955
	4956	/**
	4957	* Reads the start of a map of the exact size given.
	4958	*
	4959	* A number of values follow equal to twice the element count of the map,
	4960	* alternating between keys and values. @ref mpack_done_map() must be called
	4961	* once all elements have been read.
	4962	*
	4963	* @note Maps in JSON are unordered, so it is recommended not to expect
	4964	* a specific ordering for your map values in case your data is converted
	4965	* to/from JSON.
	4966	*
	4967	* @throws mpack_error_type if the value is not a map or if its size
	4968	* does not match the given count.
	4969	*/
	4970	void mpack_expect_map_match(mpack_reader_t* reader, uint32_t count);
	4971
	4972	/**
	4973	* Reads a nil node or the start of a map, returning whether a map was
	4974	* read and placing its number of key/value pairs in count.
	4975	*
	4976	* If a map was read, a number of values follow equal to twice the element count
	4977	* of the map, alternating between keys and values. @ref mpack_done_map() should
	4978	* also be called once all elements have been read (only if a map was read.)
	4979	*
	4980	* @note Maps in JSON are unordered, so it is recommended not to expect
	4981	* a specific ordering for your map values in case your data is converted
	4982	* to/from JSON.
	4983	*
	4984	* @warning This call is dangerous! It does not have a size limit, and it
	4985	* does not have any way of checking whether there is enough data in the
	4986	* message (since the data could be coming from a stream.) When looping
	4987	* through the map's contents, you must check for errors on each iteration
	4988	* of the loop. Otherwise an attacker could craft a message declaring a map
	4989	* of a billion elements which would throw your parsing code into an
	4990	* infinite loop! You should strongly consider using mpack_expect_map_max_or_nil()
	4991	* with a safe maximum size instead.
	4992	*
	4993	* @returns @c true if a map was read successfully; @c false if nil was read
	4994	* or an error occured.
	4995	* @throws mpack_error_type if the value is not a nil or map.
	4996	*/
	4997	bool mpack_expect_map_or_nil(mpack_reader_t* reader, uint32_t* count);
	4998
	4999	/**
	5000	* Reads a nil node or the start of a map with a number of elements at most
	5001	* max_count, returning whether a map was read and placing its number of
	5002	* key/value pairs in count.
	5003	*
	5004	* If a map was read, a number of values follow equal to twice the element count
	5005	* of the map, alternating between keys and values. @ref mpack_done_map() should
	5006	* anlso be called once all elements have been read (only if a map was read.)
	5007	*
	5008	* @note Maps in JSON are unordered, so it is recommended not to expect
	5009	* a specific ordering for your map values in case your data is converted
	5010	* to/from JSON. Consider using mpack_expect_key_cstr() or mpack_expect_key_uint()
	5011	* to switch on the key; see @ref docs/expect.md for examples.
	5012	*
	5013	* @returns @c true if a map was read successfully; @c false if nil was read
	5014	* or an error occured.
	5015	* @throws mpack_error_type if the value is not a nil or map.
	5016	*/
	5017	bool mpack_expect_map_max_or_nil(mpack_reader_t* reader, uint32_t max_count, uint32_t* count);
	5018
	5019	/**
	5020	* Reads the start of an array, returning its element count.
	5021	*
	5022	* A number of values follow equal to the element count of the array.
	5023	* @ref mpack_done_array() must be called once all elements have been read.
	5024	*
	5025	* @warning This call is dangerous! It does not have a size limit, and it
	5026	* does not have any way of checking whether there is enough data in the
	5027	* message (since the data could be coming from a stream.) When looping
	5028	* through the array's contents, you must check for errors on each iteration
	5029	* of the loop. Otherwise an attacker could craft a message declaring an array
	5030	* of a billion elements which would throw your parsing code into an
	5031	* infinite loop! You should strongly consider using mpack_expect_array_max()
	5032	* with a safe maximum size instead.
	5033	*/
	5034	uint32_t mpack_expect_array(mpack_reader_t* reader);
	5035
	5036	/**
	5037	* Reads the start of an array with a number of elements in the given range,
	5038	* returning its element count.
	5039	*
	5040	* A number of values follow equal to the element count of the array.
	5041	* @ref mpack_done_array() must be called once all elements have been read.
	5042	*
	5043	* min_count is returned if an error occurs.
	5044	*
	5045	* @throws mpack_error_type if the value is not an array or if its size does
	5046	* not fall within the given range.
	5047	*/
	5048	uint32_t mpack_expect_array_range(mpack_reader_t* reader, uint32_t min_count, uint32_t max_count);
	5049
	5050	/**
	5051	* Reads the start of an array with a number of elements at most @a max_count,
	5052	* returning its element count.
	5053	*
	5054	* A number of values follow equal to the element count of the array.
	5055	* @ref mpack_done_array() must be called once all elements have been read.
	5056	*
	5057	* Zero is returned if an error occurs.
	5058	*
	5059	* @throws mpack_error_type if the value is not an array or if its size is
	5060	* greater than max_count.
	5061	*/
	5062	MPACK_INLINE uint32_t mpack_expect_array_max(mpack_reader_t* reader, uint32_t max_count) {
	5063	return mpack_expect_array_range(reader, 0, max_count);
	5064	}
	5065
	5066	/**
	5067	* Reads the start of an array of the exact size given.
	5068	*
	5069	* A number of values follow equal to the element count of the array.
	5070	* @ref mpack_done_array() must be called once all elements have been read.
	5071	*
	5072	* @throws mpack_error_type if the value is not an array or if its size does
	5073	* not match the given count.
	5074	*/
	5075	void mpack_expect_array_match(mpack_reader_t* reader, uint32_t count);
	5076
	5077	/**
	5078	* Reads a nil node or the start of an array, returning whether an array was
	5079	* read and placing its number of elements in count.
	5080	*
	5081	* If an array was read, a number of values follow equal to the element count
	5082	* of the array. @ref mpack_done_array() should also be called once all elements
	5083	* have been read (only if an array was read.)
	5084	*
	5085	* @warning This call is dangerous! It does not have a size limit, and it
	5086	* does not have any way of checking whether there is enough data in the
	5087	* message (since the data could be coming from a stream.) When looping
	5088	* through the array's contents, you must check for errors on each iteration
	5089	* of the loop. Otherwise an attacker could craft a message declaring an array
	5090	* of a billion elements which would throw your parsing code into an
	5091	* infinite loop! You should strongly consider using mpack_expect_array_max_or_nil()
	5092	* with a safe maximum size instead.
	5093	*
	5094	* @returns @c true if an array was read successfully; @c false if nil was read
	5095	* or an error occured.
	5096	* @throws mpack_error_type if the value is not a nil or array.
	5097	*/
	5098	bool mpack_expect_array_or_nil(mpack_reader_t* reader, uint32_t* count);
	5099
	5100	/**
	5101	* Reads a nil node or the start of an array with a number of elements at most
	5102	* max_count, returning whether an array was read and placing its number of
	5103	* key/value pairs in count.
	5104	*
	5105	* If an array was read, a number of values follow equal to the element count
	5106	* of the array. @ref mpack_done_array() should also be called once all elements
	5107	* have been read (only if an array was read.)
	5108	*
	5109	* @returns @c true if an array was read successfully; @c false if nil was read
	5110	* or an error occured.
	5111	* @throws mpack_error_type if the value is not a nil or array.
	5112	*/
	5113	bool mpack_expect_array_max_or_nil(mpack_reader_t* reader, uint32_t max_count, uint32_t* count);
	5114
	5115	#ifdef MPACK_MALLOC
	5116	/**
	5117	* @hideinitializer
	5118	*
	5119	* Reads the start of an array and allocates storage for it, placing its
	5120	* size in out_count. A number of objects follow equal to the element count
	5121	* of the array. You must call @ref mpack_done_array() when done (even
	5122	* if the element count is zero.)
	5123	*
	5124	* If an error occurs, NULL is returned and the reader is placed in an
	5125	* error state.
	5126	*
	5127	* If the count is zero, NULL is returned. This does not indicate error.
	5128	* You should not check the return value for NULL to check for errors; only
	5129	* check the reader's error state.
	5130	*
	5131	* The allocated array must be freed with MPACK_FREE() (or simply free()
	5132	* if MPack's allocator hasn't been customized.)
	5133	*
	5134	* @throws mpack_error_type if the value is not an array or if its size is
	5135	* greater than max_count.
	5136	*/
	5137	#define mpack_expect_array_alloc(reader, Type, max_count, out_count) \
	5138	((Type*)mpack_expect_array_alloc_impl(reader, sizeof(Type), max_count, out_count, false))
	5139
	5140	/**
	5141	* @hideinitializer
	5142	*
	5143	* Reads a nil node or the start of an array and allocates storage for it,
	5144	* placing its size in out_count. A number of objects follow equal to the element
	5145	* count of the array if a non-empty array was read.
	5146	*
	5147	* If an error occurs, NULL is returned and the reader is placed in an
	5148	* error state.
	5149	*
	5150	* If a nil node was read, NULL is returned. If an empty array was read,
	5151	* mpack_done_array() is called automatically and NULL is returned. These
	5152	* do not indicate error. You should not check the return value for NULL
	5153	* to check for errors; only check the reader's error state.
	5154	*
	5155	* The allocated array must be freed with MPACK_FREE() (or simply free()
	5156	* if MPack's allocator hasn't been customized.)
	5157	*
	5158	* @warning You must call @ref mpack_done_array() if and only if a non-zero
	5159	* element count is read. This function does not differentiate between nil
	5160	* and an empty array.
	5161	*
	5162	* @throws mpack_error_type if the value is not an array or if its size is
	5163	* greater than max_count.
	5164	*/
	5165	#define mpack_expect_array_or_nil_alloc(reader, Type, max_count, out_count) \
	5166	((Type*)mpack_expect_array_alloc_impl(reader, sizeof(Type), max_count, out_count, true))
	5167	#endif
	5168
	5169	/**
	5170	* @}
	5171	*/
	5172
	5173	/** @cond */
	5174	#ifdef MPACK_MALLOC
	5175	void* mpack_expect_array_alloc_impl(mpack_reader_t* reader,
	5176	size_t element_size, uint32_t max_count, uint32_t* out_count, bool allow_nil);
	5177	#endif
	5178	/** @endcond */
	5179
	5180
	5181	/**
	5182	* @name String Functions
	5183	* @{
	5184	*/
	5185
	5186	/**
	5187	* Reads the start of a string, returning its size in bytes.
	5188	*
	5189	* The bytes follow and must be read separately with mpack_read_bytes()
	5190	* or mpack_read_bytes_inplace(). mpack_done_str() must be called
	5191	* once all bytes have been read.
	5192	*
	5193	* NUL bytes are allowed in the string, and no encoding checks are done.
	5194	*
	5195	* mpack_error_type is raised if the value is not a string.
	5196	*/
	5197	uint32_t mpack_expect_str(mpack_reader_t* reader);
	5198
	5199	/**
	5200	* Reads a string of at most the given size, writing it into the
	5201	* given buffer and returning its size in bytes.
	5202	*
	5203	* This does not add a null-terminator! Use mpack_expect_cstr() to
	5204	* add a null-terminator.
	5205	*
	5206	* NUL bytes are allowed in the string, and no encoding checks are done.
	5207	*/
	5208	size_t mpack_expect_str_buf(mpack_reader_t* reader, char* buf, size_t bufsize);
	5209
	5210	/**
	5211	* Reads a string into the given buffer, ensuring it is a valid UTF-8 string
	5212	* and returning its size in bytes.
	5213	*
	5214	* This does not add a null-terminator! Use mpack_expect_utf8_cstr() to
	5215	* add a null-terminator.
	5216	*
	5217	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	5218	* WTF-8. Only pure UTF-8 is allowed.
	5219	*
	5220	* NUL bytes are allowed in the string (as they are in UTF-8.)
	5221	*
	5222	* Raises mpack_error_too_big if there is not enough room for the string.
	5223	* Raises mpack_error_type if the value is not a string or is not a valid UTF-8 string.
	5224	*/
	5225	size_t mpack_expect_utf8(mpack_reader_t* reader, char* buf, size_t bufsize);
	5226
	5227	/**
	5228	* Reads the start of a string, raising an error if its length is not
	5229	* at most the given number of bytes (not including any null-terminator.)
	5230	*
	5231	* The bytes follow and must be read separately with mpack_read_bytes()
	5232	* or mpack_read_bytes_inplace(). @ref mpack_done_str() must be called
	5233	* once all bytes have been read.
	5234	*
	5235	* @throws mpack_error_type If the value is not a string.
	5236	* @throws mpack_error_too_big If the string's length in bytes is larger than the given maximum size.
	5237	*/
	5238	MPACK_INLINE uint32_t mpack_expect_str_max(mpack_reader_t* reader, uint32_t maxsize) {
	5239	uint32_t length = mpack_expect_str(reader);
	5240	if (length > maxsize) {
	5241	mpack_reader_flag_error(reader, mpack_error_too_big);
	5242	return 0;
	5243	}
	5244	return length;
	5245	}
	5246
	5247	/**
	5248	* Reads the start of a string, raising an error if its length is not
	5249	* exactly the given number of bytes (not including any null-terminator.)
	5250	*
	5251	* The bytes follow and must be read separately with mpack_read_bytes()
	5252	* or mpack_read_bytes_inplace(). @ref mpack_done_str() must be called
	5253	* once all bytes have been read.
	5254	*
	5255	* mpack_error_type is raised if the value is not a string or if its
	5256	* length does not match.
	5257	*/
	5258	MPACK_INLINE void mpack_expect_str_length(mpack_reader_t* reader, uint32_t count) {
	5259	if (mpack_expect_str(reader) != count)
	5260	mpack_reader_flag_error(reader, mpack_error_type);
	5261	}
	5262
	5263	/**
	5264	* Reads a string, ensuring it exactly matches the given string.
	5265	*
	5266	* Remember that maps are unordered in JSON. Don't use this for map keys
	5267	* unless the map has only a single key!
	5268	*/
	5269	void mpack_expect_str_match(mpack_reader_t* reader, const char* str, size_t length);
	5270
	5271	/**
	5272	* Reads a string into the given buffer, ensures it has no null bytes,
	5273	* and adds a null-terminator at the end.
	5274	*
	5275	* Raises mpack_error_too_big if there is not enough room for the string and null-terminator.
	5276	* Raises mpack_error_type if the value is not a string or contains a null byte.
	5277	*/
	5278	void mpack_expect_cstr(mpack_reader_t* reader, char* buf, size_t size);
	5279
	5280	/**
	5281	* Reads a string into the given buffer, ensures it is a valid UTF-8 string
	5282	* without NUL characters, and adds a null-terminator at the end.
	5283	*
	5284	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	5285	* WTF-8. Only pure UTF-8 is allowed, but without the NUL character, since
	5286	* it cannot be represented in a null-terminated string.
	5287	*
	5288	* Raises mpack_error_too_big if there is not enough room for the string and null-terminator.
	5289	* Raises mpack_error_type if the value is not a string or is not a valid UTF-8 string.
	5290	*/
	5291	void mpack_expect_utf8_cstr(mpack_reader_t* reader, char* buf, size_t size);
	5292
	5293	#ifdef MPACK_MALLOC
	5294	/**
	5295	* Reads a string with the given total maximum size (including space for a
	5296	* null-terminator), allocates storage for it, ensures it has no null-bytes,
	5297	* and adds a null-terminator at the end. You assume ownership of the
	5298	* returned pointer if reading succeeds.
	5299	*
	5300	* The allocated string must be freed with MPACK_FREE() (or simply free()
	5301	* if MPack's allocator hasn't been customized.)
	5302	*
	5303	* @throws mpack_error_too_big If the string plus null-terminator is larger than the given maxsize.
	5304	* @throws mpack_error_type If the value is not a string or contains a null byte.
	5305	*/
	5306	char* mpack_expect_cstr_alloc(mpack_reader_t* reader, size_t maxsize);
	5307
	5308	/**
	5309	* Reads a string with the given total maximum size (including space for a
	5310	* null-terminator), allocates storage for it, ensures it is valid UTF-8
	5311	* with no null-bytes, and adds a null-terminator at the end. You assume
	5312	* ownership of the returned pointer if reading succeeds.
	5313	*
	5314	* The length in bytes of the string, not including the null-terminator,
	5315	* will be written to size.
	5316	*
	5317	* This does not accept any UTF-8 variant such as Modified UTF-8, CESU-8 or
	5318	* WTF-8. Only pure UTF-8 is allowed, but without the NUL character, since
	5319	* it cannot be represented in a null-terminated string.
	5320	*
	5321	* The allocated string must be freed with MPACK_FREE() (or simply free()
	5322	* if MPack's allocator hasn't been customized.)
	5323	* if you want a null-terminator.
	5324	*
	5325	* @throws mpack_error_too_big If the string plus null-terminator is larger
	5326	* than the given maxsize.
	5327	* @throws mpack_error_type If the value is not a string or contains
	5328	* invalid UTF-8 or a null byte.
	5329	*/
	5330	char* mpack_expect_utf8_cstr_alloc(mpack_reader_t* reader, size_t maxsize);
	5331	#endif
	5332
	5333	/**
	5334	* Reads a string, ensuring it exactly matches the given null-terminated
	5335	* string.
	5336	*
	5337	* Remember that maps are unordered in JSON. Don't use this for map keys
	5338	* unless the map has only a single key!
	5339	*/
	5340	MPACK_INLINE void mpack_expect_cstr_match(mpack_reader_t* reader, const char* cstr) {
	5341	mpack_assert(cstr != NULL, "cstr pointer is NULL");
	5342	mpack_expect_str_match(reader, cstr, mpack_strlen(cstr));
	5343	}
	5344
	5345	/**
	5346	* @}
	5347	*/
	5348
	5349	/**
	5350	* @name Binary Data
	5351	* @{
	5352	*/
	5353
	5354	/**
	5355	* Reads the start of a binary blob, returning its size in bytes.
	5356	*
	5357	* The bytes follow and must be read separately with mpack_read_bytes()
	5358	* or mpack_read_bytes_inplace(). @ref mpack_done_bin() must be called
	5359	* once all bytes have been read.
	5360	*
	5361	* mpack_error_type is raised if the value is not a binary blob.
	5362	*/
	5363	uint32_t mpack_expect_bin(mpack_reader_t* reader);
	5364
	5365	/**
	5366	* Reads the start of a binary blob, raising an error if its length is not
	5367	* at most the given number of bytes.
	5368	*
	5369	* The bytes follow and must be read separately with mpack_read_bytes()
	5370	* or mpack_read_bytes_inplace(). @ref mpack_done_bin() must be called
	5371	* once all bytes have been read.
	5372	*
	5373	* mpack_error_type is raised if the value is not a binary blob or if its
	5374	* length does not match.
	5375	*/
	5376	MPACK_INLINE uint32_t mpack_expect_bin_max(mpack_reader_t* reader, uint32_t maxsize) {
	5377	uint32_t length = mpack_expect_bin(reader);
	5378	if (length > maxsize) {
	5379	mpack_reader_flag_error(reader, mpack_error_type);
	5380	return 0;
	5381	}
	5382	return length;
	5383	}
	5384
	5385	/**
	5386	* Reads the start of a binary blob, raising an error if its length is not
	5387	* exactly the given number of bytes.
	5388	*
	5389	* The bytes follow and must be read separately with mpack_read_bytes()
	5390	* or mpack_read_bytes_inplace(). @ref mpack_done_bin() must be called
	5391	* once all bytes have been read.
	5392	*
	5393	* mpack_error_type is raised if the value is not a binary blob or if its
	5394	* length does not match.
	5395	*/
	5396	MPACK_INLINE void mpack_expect_bin_size(mpack_reader_t* reader, uint32_t count) {
	5397	if (mpack_expect_bin(reader) != count)
	5398	mpack_reader_flag_error(reader, mpack_error_type);
	5399	}
	5400
	5401	/**
	5402	* Reads a binary blob into the given buffer, returning its size in bytes.
	5403	*
	5404	* For compatibility, this will accept if the underlying type is string or
	5405	* binary (since in MessagePack 1.0, strings and binary data were combined
	5406	* under the "raw" type which became string in 1.1.)
	5407	*/
	5408	size_t mpack_expect_bin_buf(mpack_reader_t* reader, char* buf, size_t size);
	5409
	5410	/**
	5411	* Reads a binary blob with the given total maximum size, allocating storage for it.
	5412	*/
	5413	char* mpack_expect_bin_alloc(mpack_reader_t* reader, size_t maxsize, size_t* size);
	5414
	5415	/**
	5416	* @}
	5417	*/
	5418
	5419	/**
	5420	* @name Extension Functions
	5421	* @{
	5422	*/
	5423
	5424	#if MPACK_EXTENSIONS
	5425	/**
	5426	* Reads the start of an extension blob, returning its size in bytes and
	5427	* placing the type into @p type.
	5428	*
	5429	* The bytes follow and must be read separately with mpack_read_bytes()
	5430	* or mpack_read_bytes_inplace(). @ref mpack_done_ext() must be called
	5431	* once all bytes have been read.
	5432	*
	5433	* @p type will be a user-defined type in the range [0,127] or a reserved type
	5434	* in the range [-128,-2].
	5435	*
	5436	* mpack_error_type is raised if the value is not an extension blob. The @p
	5437	* type value is zero if an error occurs.
	5438	*
	5439	* @note This cannot be used to match a timestamp. @ref mpack_error_type will
	5440	* be flagged if the value is a timestamp. Use mpack_expect_timestamp() or
	5441	* mpack_expect_timestamp_truncate() instead.
	5442	*
	5443	* @note This requires @ref MPACK_EXTENSIONS.
	5444	*
	5445	* @warning Be careful when using reserved types. They may no longer be ext
	5446	* types in the future, and previously valid data containing reserved types may
	5447	* become invalid in the future.
	5448	*/
	5449	uint32_t mpack_expect_ext(mpack_reader_t* reader, int8_t* type);
	5450
	5451	/**
	5452	* Reads the start of an extension blob, raising an error if its length is not
	5453	* at most the given number of bytes and placing the type into @p type.
	5454	*
	5455	* The bytes follow and must be read separately with mpack_read_bytes()
	5456	* or mpack_read_bytes_inplace(). @ref mpack_done_ext() must be called
	5457	* once all bytes have been read.
	5458	*
	5459	* mpack_error_type is raised if the value is not an extension blob or if its
	5460	* length does not match. The @p type value is zero if an error is raised.
	5461	*
	5462	* @p type will be a user-defined type in the range [0,127] or a reserved type
	5463	* in the range [-128,-2].
	5464	*
	5465	* @note This cannot be used to match a timestamp. @ref mpack_error_type will
	5466	* be flagged if the value is a timestamp. Use mpack_expect_timestamp() or
	5467	* mpack_expect_timestamp_truncate() instead.
	5468	*
	5469	* @note This requires @ref MPACK_EXTENSIONS.
	5470	*
	5471	* @warning Be careful when using reserved types. They may no longer be ext
	5472	* types in the future, and previously valid data containing reserved types may
	5473	* become invalid in the future.
	5474	*
	5475	* @see mpack_expect_ext()
	5476	*/
	5477	MPACK_INLINE uint32_t mpack_expect_ext_max(mpack_reader_t* reader, int8_t* type, uint32_t maxsize) {
	5478	uint32_t length = mpack_expect_ext(reader, type);
	5479	if (length > maxsize) {
	5480	mpack_reader_flag_error(reader, mpack_error_type);
	5481	return 0;
	5482	}
	5483	return length;
	5484	}
	5485
	5486	/**
	5487	* Reads the start of an extension blob, raising an error if its length is not
	5488	* exactly the given number of bytes and placing the type into @p type.
	5489	*
	5490	* The bytes follow and must be read separately with mpack_read_bytes()
	5491	* or mpack_read_bytes_inplace(). @ref mpack_done_ext() must be called
	5492	* once all bytes have been read.
	5493	*
	5494	* mpack_error_type is raised if the value is not an extension blob or if its
	5495	* length does not match. The @p type value is zero if an error is raised.
	5496	*
	5497	* @p type will be a user-defined type in the range [0,127] or a reserved type
	5498	* in the range [-128,-2].
	5499	*
	5500	* @note This cannot be used to match a timestamp. @ref mpack_error_type will
	5501	* be flagged if the value is a timestamp. Use mpack_expect_timestamp() or
	5502	* mpack_expect_timestamp_truncate() instead.
	5503	*
	5504	* @note This requires @ref MPACK_EXTENSIONS.
	5505	*
	5506	* @warning Be careful when using reserved types. They may no longer be ext
	5507	* types in the future, and previously valid data containing reserved types may
	5508	* become invalid in the future.
	5509	*
	5510	* @see mpack_expect_ext()
	5511	*/
	5512	MPACK_INLINE void mpack_expect_ext_size(mpack_reader_t* reader, int8_t* type, uint32_t count) {
	5513	if (mpack_expect_ext(reader, type) != count) {
	5514	*type = 0;
	5515	mpack_reader_flag_error(reader, mpack_error_type);
	5516	}
	5517	}
	5518
	5519	/**
	5520	* Reads an extension blob into the given buffer, returning its size in bytes
	5521	* and placing the type into @p type.
	5522	*
	5523	* mpack_error_type is raised if the value is not an extension blob or if its
	5524	* length does not match. The @p type value is zero if an error is raised.
	5525	*
	5526	* @p type will be a user-defined type in the range [0,127] or a reserved type
	5527	* in the range [-128,-2].
	5528	*
	5529	* @note This cannot be used to match a timestamp. @ref mpack_error_type will
	5530	* be flagged if the value is a timestamp. Use mpack_expect_timestamp() or
	5531	* mpack_expect_timestamp_truncate() instead.
	5532	*
	5533	* @warning Be careful when using reserved types. They may no longer be ext
	5534	* types in the future, and previously valid data containing reserved types may
	5535	* become invalid in the future.
	5536	*
	5537	* @note This requires @ref MPACK_EXTENSIONS.
	5538	*
	5539	* @see mpack_expect_ext()
	5540	*/
	5541	size_t mpack_expect_ext_buf(mpack_reader_t* reader, int8_t* type, char* buf, size_t size);
	5542	#endif
	5543
	5544	#if MPACK_EXTENSIONS && defined(MPACK_MALLOC)
	5545	/**
	5546	* Reads an extension blob with the given total maximum size, allocating
	5547	* storage for it, and placing the type into @p type.
	5548	*
	5549	* mpack_error_type is raised if the value is not an extension blob or if its
	5550	* length does not match. The @p type value is zero if an error is raised.
	5551	*
	5552	* @p type will be a user-defined type in the range [0,127] or a reserved type
	5553	* in the range [-128,-2].
	5554	*
	5555	* @note This cannot be used to match a timestamp. @ref mpack_error_type will
	5556	* be flagged if the value is a timestamp. Use mpack_expect_timestamp() or
	5557	* mpack_expect_timestamp_truncate() instead.
	5558	*
	5559	* @warning Be careful when using reserved types. They may no longer be ext
	5560	* types in the future, and previously valid data containing reserved types may
	5561	* become invalid in the future.
	5562	*
	5563	* @note This requires @ref MPACK_EXTENSIONS and @ref MPACK_MALLOC.
	5564	*
	5565	* @see mpack_expect_ext()
	5566	*/
	5567	char* mpack_expect_ext_alloc(mpack_reader_t* reader, int8_t* type, size_t maxsize, size_t* size);
	5568	#endif
	5569
	5570	/**
	5571	* @}
	5572	*/
	5573
	5574	/**
	5575	* @name Special Functions
	5576	* @{
	5577	*/
	5578
	5579	/**
	5580	* Reads a MessagePack object header (an MPack tag), expecting it to exactly
	5581	* match the given tag.
	5582	*
	5583	* If the type is compound (i.e. is a map, array, string, binary or
	5584	* extension type), additional reads are required to get the contained
	5585	* data, and the corresponding done function must be called when done.
	5586	*
	5587	* @throws mpack_error_type if the tag does not match
	5588	*
	5589	* @see mpack_read_bytes()
	5590	* @see mpack_done_array()
	5591	* @see mpack_done_map()
	5592	* @see mpack_done_str()
	5593	* @see mpack_done_bin()
	5594	* @see mpack_done_ext()
	5595	*/
	5596	void mpack_expect_tag(mpack_reader_t* reader, mpack_tag_t tag);
	5597
	5598	/**
	5599	* Expects a string matching one of the strings in the given array,
	5600	* returning its array index.
	5601	*
	5602	* If the value does not match any of the given strings,
	5603	* @ref mpack_error_type is flagged. Use mpack_expect_enum_optional()
	5604	* if you want to allow other values than the given strings.
	5605	*
	5606	* If any error occurs or the reader is in an error state, @a count
	5607	* is returned.
	5608	*
	5609	* This can be used to quickly parse a string into an enum when the
	5610	* enum values range from 0 to @a count-1. If the last value in the
	5611	* enum is a special "count" value, it can be passed as the count,
	5612	* and the return value can be cast directly to the enum type.
	5613	*
	5614	* @code{.c}
	5615	* typedef enum { APPLE , BANANA , ORANGE , COUNT} fruit_t;
	5616	* const char* fruits[] = {"apple", "banana", "orange"};
	5617	*
	5618	* fruit_t fruit = (fruit_t)mpack_expect_enum(reader, fruits, COUNT);
	5619	* @endcode
	5620	*
	5621	* See @ref docs/expect.md for more examples.
	5622	*
	5623	* The maximum string length is the size of the buffer (strings are read in-place.)
	5624	*
	5625	* @param reader The reader
	5626	* @param strings An array of expected strings of length count
	5627	* @param count The number of strings
	5628	* @return The index of the matched string, or @a count in case of error
	5629	*/
	5630	size_t mpack_expect_enum(mpack_reader_t* reader, const char* strings[], size_t count);
	5631
	5632	/**
	5633	* Expects a string matching one of the strings in the given array
	5634	* returning its array index, or @a count if no strings match.
	5635	*
	5636	* If the value is not a string, or it does not match any of the
	5637	* given strings, @a count is returned and no error is flagged.
	5638	*
	5639	* If any error occurs or the reader is in an error state, @a count
	5640	* is returned.
	5641	*
	5642	* This can be used to quickly parse a string into an enum when the
	5643	* enum values range from 0 to @a count-1. If the last value in the
	5644	* enum is a special "count" value, it can be passed as the count,
	5645	* and the return value can be cast directly to the enum type.
	5646	*
	5647	* @code{.c}
	5648	* typedef enum { APPLE , BANANA , ORANGE , COUNT} fruit_t;
	5649	* const char* fruits[] = {"apple", "banana", "orange"};
	5650	*
	5651	* fruit_t fruit = (fruit_t)mpack_expect_enum_optional(reader, fruits, COUNT);
	5652	* @endcode
	5653	*
	5654	* See @ref docs/expect.md for more examples.
	5655	*
	5656	* The maximum string length is the size of the buffer (strings are read in-place.)
	5657	*
	5658	* @param reader The reader
	5659	* @param strings An array of expected strings of length count
	5660	* @param count The number of strings
	5661	*
	5662	* @return The index of the matched string, or @a count if it does not
	5663	* match or an error occurs
	5664	*/
	5665	size_t mpack_expect_enum_optional(mpack_reader_t* reader, const char* strings[], size_t count);
	5666
	5667	/**
	5668	* Expects an unsigned integer map key between 0 and count-1, marking it
	5669	* as found in the given bool array and returning it.
	5670	*
	5671	* This is a helper for switching among int keys in a map. It is
	5672	* typically used with an enum to define the key values. It should
	5673	* be called in the expression of a switch() statement. See @ref
	5674	* docs/expect.md for an example.
	5675	*
	5676	* The found array must be cleared before expecting the first key. If the
	5677	* flag for a given key is already set when found (i.e. the map contains a
	5678	* duplicate key), mpack_error_invalid is flagged.
	5679	*
	5680	* If the key is not a non-negative integer, or if the key is @a count or
	5681	* larger, @a count is returned and no error is flagged. If you want an error
	5682	* on unrecognized keys, flag an error in the default case in your switch;
	5683	* otherwise you must call mpack_discard() to discard its content.
	5684	*
	5685	* @param reader The reader
	5686	* @param found An array of bool flags of length count
	5687	* @param count The number of values in the found array, and one more than the
	5688	* maximum allowed key
	5689	*
	5690	* @see @ref docs/expect.md
	5691	*/
	5692	size_t mpack_expect_key_uint(mpack_reader_t* reader, bool found[], size_t count);
	5693
	5694	/**
	5695	* Expects a string map key matching one of the strings in the given key list,
	5696	* marking it as found in the given bool array and returning its index.
	5697	*
	5698	* This is a helper for switching among string keys in a map. It is
	5699	* typically used with an enum with names matching the strings in the
	5700	* array to define the key indices. It should be called in the expression
	5701	* of a switch() statement. See @ref docs/expect.md for an example.
	5702	*
	5703	* The found array must be cleared before expecting the first key. If the
	5704	* flag for a given key is already set when found (i.e. the map contains a
	5705	* duplicate key), mpack_error_invalid is flagged.
	5706	*
	5707	* If the key is unrecognized, count is returned and no error is flagged. If
	5708	* you want an error on unrecognized keys, flag an error in the default case
	5709	* in your switch; otherwise you must call mpack_discard() to discard its content.
	5710	*
	5711	* The maximum key length is the size of the buffer (keys are read in-place.)
	5712	*
	5713	* @param reader The reader
	5714	* @param keys An array of expected string keys of length count
	5715	* @param found An array of bool flags of length count
	5716	* @param count The number of values in the keys and found arrays
	5717	*
	5718	* @see @ref docs/expect.md
	5719	*/
	5720	size_t mpack_expect_key_cstr(mpack_reader_t* reader, const char* keys[],
	5721	bool found[], size_t count);
	5722
	5723	/**
	5724	* @}
	5725	*/
	5726
	5727	/**
	5728	* @}
	5729	*/
	5730
	5731	#endif
	5732
	5733	MPACK_HEADER_END
	5734
	5735	#endif
	5736
	5737
	5738
	5739	/* mpack/mpack-node.h.h */
	5740
	5741	/**
	5742	* @file
	5743	*
	5744	* Declares the MPack dynamic Node API.
	5745	*/
	5746
	5747	#ifndef MPACK_NODE_H
	5748	#define MPACK_NODE_H 1
	5749
	5750	/* #include "mpack-reader.h" */
	5751
	5752	MPACK_HEADER_START
	5753
	5754	#if MPACK_NODE
	5755
	5756	/**
	5757	* @defgroup node Node API
	5758	*
	5759	* The MPack Node API allows you to parse a chunk of MessagePack into a
	5760	* dynamically typed data structure, providing random access to the parsed
	5761	* data.
	5762	*
	5763	* See @ref docs/node.md for examples.
	5764	*
	5765	* @{
	5766	*/
	5767
	5768	/**
	5769	* A handle to node data in a parsed MPack tree.
	5770	*
	5771	* Nodes represent either primitive values or compound types. If a
	5772	* node is a compound type, it contains a pointer to its child nodes,
	5773	* or a pointer to its underlying data.
	5774	*
	5775	* Nodes are immutable.
	5776	*
	5777	* @note @ref mpack_node_t is an opaque reference to the node data, not the
	5778	* node data itself. (It contains pointers to both the node data and the tree.)
	5779	* It is passed by value in the Node API.
	5780	*/
	5781	typedef struct mpack_node_t mpack_node_t;
	5782
	5783	/**
	5784	* The storage for nodes in an MPack tree.
	5785	*
	5786	* You only need to use this if you intend to provide your own storage
	5787	* for nodes instead of letting the tree allocate it.
	5788	*
	5789	* @ref mpack_node_data_t is 16 bytes on most common architectures (32-bit
	5790	* and 64-bit.)
	5791	*/
	5792	typedef struct mpack_node_data_t mpack_node_data_t;
	5793
	5794	/**
	5795	* An MPack tree parser to parse a blob or stream of MessagePack.
	5796	*
	5797	* When a message is parsed, the tree contains a single root node which
	5798	* contains all parsed data. The tree and its nodes are immutable.
	5799	*/
	5800	typedef struct mpack_tree_t mpack_tree_t;
	5801
	5802	/**
	5803	* An error handler function to be called when an error is flagged on
	5804	* the tree.
	5805	*
	5806	* The error handler will only be called once on the first error flagged;
	5807	* any subsequent node reads and errors are ignored, and the tree is
	5808	* permanently in that error state.
	5809	*
	5810	* MPack is safe against non-local jumps out of error handler callbacks.
	5811	* This means you are allowed to longjmp or throw an exception (in C++,
	5812	* Objective-C, or with SEH) out of this callback.
	5813	*
	5814	* Bear in mind when using longjmp that local non-volatile variables that
	5815	* have changed are undefined when setjmp() returns, so you can't put the
	5816	* tree on the stack in the same activation frame as the setjmp without
	5817	* declaring it volatile.
	5818	*
	5819	* You must still eventually destroy the tree. It is not destroyed
	5820	* automatically when an error is flagged. It is safe to destroy the
	5821	* tree within this error callback, but you will either need to perform
	5822	* a non-local jump, or store something in your context to identify
	5823	* that the tree is destroyed since any future accesses to it cause
	5824	* undefined behavior.
	5825	*/
	5826	typedef void (mpack_tree_error_t)(mpack_tree_t tree, mpack_error_t error);
	5827
	5828	/**
	5829	* The MPack tree's read function. It should fill the buffer with as many bytes
	5830	* as are immediately available up to the given @c count, returning the number
	5831	* of bytes written to the buffer.
	5832	*
	5833	* In case of error, it should flag an appropriate error on the reader
	5834	* (usually @ref mpack_error_io.)
	5835	*
	5836	* The blocking or non-blocking behaviour of the read should match whether you
	5837	* are using mpack_tree_parse() or mpack_tree_try_parse().
	5838	*
	5839	* If you are using mpack_tree_parse(), the read should block until at least
	5840	* one byte is read. If you return 0, mpack_tree_parse() will raise @ref
	5841	* mpack_error_io.
	5842	*
	5843	* If you are using mpack_tree_try_parse(), the read function can always
	5844	* return 0, and must never block waiting for data (otherwise
	5845	* mpack_tree_try_parse() would be equivalent to mpack_tree_parse().)
	5846	* When you return 0, mpack_tree_try_parse() will return false without flagging
	5847	* an error.
	5848	*/
	5849	typedef size_t (mpack_tree_read_t)(mpack_tree_t tree, char* buffer, size_t count);
	5850
	5851	/**
	5852	* A teardown function to be called when the tree is destroyed.
	5853	*/
	5854	typedef void (mpack_tree_teardown_t)(mpack_tree_t tree);
	5855
	5856
	5857
	5858	/* Hide internals from documentation */
	5859	/** @cond */
	5860
	5861	struct mpack_node_t {
	5862	mpack_node_data_t* data;
	5863	mpack_tree_t* tree;
	5864	};
	5865
	5866	struct mpack_node_data_t {
	5867	mpack_type_t type;
	5868
	5869	/*
	5870	* The element count if the type is an array;
	5871	* the number of key/value pairs if the type is map;
	5872	* or the number of bytes if the type is str, bin or ext.
	5873	*/
	5874	uint32_t len;
	5875
	5876	union
	5877	{
	5878	bool b; /* The value if the type is bool. */
	5879	float f; /* The value if the type is float. */
	5880	double d; /* The value if the type is double. */
	5881	int64_t i; /* The value if the type is signed int. */
	5882	uint64_t u; /* The value if the type is unsigned int. */
	5883	size_t offset; /* The byte offset for str, bin and ext */
	5884	mpack_node_data_t* children; /* The children for map or array */
	5885	} value;
	5886	};
	5887
	5888	typedef struct mpack_tree_page_t {
	5889	struct mpack_tree_page_t* next;
	5890	mpack_node_data_t nodes[1]; // variable size
	5891	} mpack_tree_page_t;
	5892
	5893	typedef enum mpack_tree_parse_state_t {
	5894	mpack_tree_parse_state_not_started,
	5895	mpack_tree_parse_state_in_progress,
	5896	mpack_tree_parse_state_parsed,
	5897	} mpack_tree_parse_state_t;
	5898
	5899	typedef struct mpack_level_t {
	5900	mpack_node_data_t* child;
	5901	size_t left; // children left in level
	5902	} mpack_level_t;
	5903
	5904	typedef struct mpack_tree_parser_t {
	5905	mpack_tree_parse_state_t state;
	5906
	5907	// We keep track of the number of "possible nodes" left in the data rather
	5908	// than the number of bytes.
	5909	//
	5910	// When a map or array is parsed, we ensure at least one byte for each child
	5911	// exists and subtract them right away. This ensures that if ever a map or
	5912	// array declares more elements than could possibly be contained in the data,
	5913	// we will error out immediately rather than allocating storage for them.
	5914	//
	5915	// For example malicious data that repeats 0xDE 0xFF 0xFF (start of a map
	5916	// with 65536 key-value pairs) would otherwise cause us to run out of
	5917	// memory. With this, the parser can allocate at most as many nodes as
	5918	// there are bytes in the data (plus the paging overhead, 12%.) An error
	5919	// will be flagged immediately if and when there isn't enough data left to
	5920	// fully read all children of all open compound types on the parsing stack.
	5921	//
	5922	// Once an entire message has been parsed (and there are no nodes left to
	5923	// parse whose bytes have been subtracted), this matches the number of left
	5924	// over bytes in the data.
	5925	size_t possible_nodes_left;
	5926
	5927	mpack_node_data_t* nodes; // next node in current page/pool
	5928	size_t nodes_left; // nodes left in current page/pool
	5929
	5930	size_t current_node_reserved;
	5931	size_t level;
	5932
	5933	#ifdef MPACK_MALLOC
	5934	// It's much faster to allocate the initial parsing stack inline within the
	5935	// parser. We replace it with a heap allocation if we need to grow it.
	5936	mpack_level_t* stack;
	5937	size_t stack_capacity;
	5938	bool stack_owned;
	5939	mpack_level_t stack_local[MPACK_NODE_INITIAL_DEPTH];
	5940	#else
	5941	// Without malloc(), we have to reserve a parsing stack the maximum allowed
	5942	// parsing depth.
	5943	mpack_level_t stack[MPACK_NODE_MAX_DEPTH_WITHOUT_MALLOC];
	5944	#endif
	5945	} mpack_tree_parser_t;
	5946
	5947	struct mpack_tree_t {
	5948	mpack_tree_error_t error_fn; /* Function to call on error */
	5949	mpack_tree_read_t read_fn; /* Function to call to read more data */
	5950	mpack_tree_teardown_t teardown; /* Function to teardown the context on destroy */
	5951	void* context; /* Context for tree callbacks */
	5952
	5953	mpack_node_data_t nil_node; /* a nil node to be returned in case of error */
	5954	mpack_node_data_t missing_node; /* a missing node to be returned in optional lookups */
	5955	mpack_error_t error;
	5956
	5957	#ifdef MPACK_MALLOC
	5958	char* buffer;
	5959	size_t buffer_capacity;
	5960	#endif
	5961
	5962	const char* data;
	5963	size_t data_length; // length of data (and content of buffer, if used)
	5964
	5965	size_t size; // size in bytes of tree (usually matches data_length, but not if tree has trailing data)
	5966	size_t node_count; // total number of nodes in tree (across all pages)
	5967
	5968	size_t max_size; // maximum message size
	5969	size_t max_nodes; // maximum nodes in a message
	5970
	5971	mpack_tree_parser_t parser;
	5972	mpack_node_data_t* root;
	5973
	5974	mpack_node_data_t* pool; // pool, or NULL if no pool provided
	5975	size_t pool_count;
	5976
	5977	#ifdef MPACK_MALLOC
	5978	mpack_tree_page_t* next;
	5979	#endif
	5980	};
	5981
	5982	// internal functions
	5983
	5984	MPACK_INLINE mpack_node_t mpack_node(mpack_tree_t* tree, mpack_node_data_t* data) {
	5985	mpack_node_t node;
	5986	node.data = data;
	5987	node.tree = tree;
	5988	return node;
	5989	}
	5990
	5991	MPACK_INLINE mpack_node_data_t* mpack_node_child(mpack_node_t node, size_t child) {
	5992	return node.data->value.children + child;
	5993	}
	5994
	5995	MPACK_INLINE mpack_node_t mpack_tree_nil_node(mpack_tree_t* tree) {
	5996	return mpack_node(tree, &tree->nil_node);
	5997	}
	5998
	5999	MPACK_INLINE mpack_node_t mpack_tree_missing_node(mpack_tree_t* tree) {
	6000	return mpack_node(tree, &tree->missing_node);
	6001	}
	6002
	6003	/** @endcond */
	6004
	6005
	6006
	6007	/**
	6008	* @name Tree Initialization
	6009	* @{
	6010	*/
	6011
	6012	#ifdef MPACK_MALLOC
	6013	/**
	6014	* Initializes a tree parser with the given data.
	6015	*
	6016	* Configure the tree if desired, then call mpack_tree_parse() to parse it. The
	6017	* tree will allocate pages of nodes as needed and will free them when
	6018	* destroyed.
	6019	*
	6020	* The tree must be destroyed with mpack_tree_destroy().
	6021	*
	6022	* Any string or blob data types reference the original data, so the given data
	6023	* pointer must remain valid until after the tree is destroyed.
	6024	*/
	6025	void mpack_tree_init_data(mpack_tree_t* tree, const char* data, size_t length);
	6026
	6027	/**
	6028	* Deprecated.
	6029	*
	6030	* \deprecated Renamed to mpack_tree_init_data().
	6031	*/
	6032	MPACK_INLINE void mpack_tree_init(mpack_tree_t* tree, const char* data, size_t length) {
	6033	mpack_tree_init_data(tree, data, length);
	6034	}
	6035
	6036	/**
	6037	* Initializes a tree parser from an unbounded stream, or a stream of
	6038	* unknown length.
	6039	*
	6040	* The parser can be used to read a single message from a stream of unknown
	6041	* length, or multiple messages from an unbounded stream, allowing it to
	6042	* be used for RPC communication. Call @ref mpack_tree_parse() to parse
	6043	* a message from a blocking stream, or @ref mpack_tree_try_parse() for a
	6044	* non-blocking stream.
	6045	*
	6046	* The stream will use a growable internal buffer to store the most recent
	6047	* message, as well as allocated pages of nodes for the parse tree.
	6048	*
	6049	* Maximum allowances for message size and node count must be specified in this
	6050	* function (since the stream is unbounded.) They can be changed later with
	6051	* @ref mpack_tree_set_limits().
	6052	*
	6053	* @param tree The tree parser
	6054	* @param read_fn The read function
	6055	* @param context The context for the read function
	6056	* @param max_message_size The maximum size of a message in bytes
	6057	* @param max_message_nodes The maximum number of nodes per message. See
	6058	* @ref mpack_node_data_t for the size of nodes.
	6059	*
	6060	* @see mpack_tree_read_t
	6061	* @see mpack_reader_context()
	6062	*/
	6063	void mpack_tree_init_stream(mpack_tree_t* tree, mpack_tree_read_t read_fn, void* context,
	6064	size_t max_message_size, size_t max_message_nodes);
	6065	#endif
	6066
	6067	/**
	6068	* Initializes a tree parser with the given data, using the given node data
	6069	* pool to store the results.
	6070	*
	6071	* Configure the tree if desired, then call mpack_tree_parse() to parse it.
	6072	*
	6073	* If the data does not fit in the pool, @ref mpack_error_too_big will be flagged
	6074	* on the tree.
	6075	*
	6076	* The tree must be destroyed with mpack_tree_destroy(), even if parsing fails.
	6077	*/
	6078	void mpack_tree_init_pool(mpack_tree_t* tree, const char* data, size_t length,
	6079	mpack_node_data_t* node_pool, size_t node_pool_count);
	6080
	6081	/**
	6082	* Initializes an MPack tree directly into an error state. Use this if you
	6083	* are writing a wrapper to another <tt>mpack_tree_init*()</tt> function which
	6084	* can fail its setup.
	6085	*/
	6086	void mpack_tree_init_error(mpack_tree_t* tree, mpack_error_t error);
	6087
	6088	#if MPACK_STDIO
	6089	/**
	6090	* Initializes a tree to parse the given file. The tree must be destroyed with
	6091	* mpack_tree_destroy(), even if parsing fails.
	6092	*
	6093	* The file is opened, loaded fully into memory, and closed before this call
	6094	* returns.
	6095	*
	6096	* @param tree The tree to initialize
	6097	* @param filename The filename passed to fopen() to read the file
	6098	* @param max_bytes The maximum size of file to load, or 0 for unlimited size.
	6099	*/
	6100	void mpack_tree_init_filename(mpack_tree_t* tree, const char* filename, size_t max_bytes);
	6101
	6102	/**
	6103	* Deprecated.
	6104	*
	6105	* \deprecated Renamed to mpack_tree_init_filename().
	6106	*/
	6107	MPACK_INLINE void mpack_tree_init_file(mpack_tree_t* tree, const char* filename, size_t max_bytes) {
	6108	mpack_tree_init_filename(tree, filename, max_bytes);
	6109	}
	6110
	6111	/**
	6112	* Initializes a tree to parse the given libc FILE. This can be used to
	6113	* read from stdin, or from a file opened separately.
	6114	*
	6115	* The tree must be destroyed with mpack_tree_destroy(), even if parsing fails.
	6116	*
	6117	* The FILE is fully loaded fully into memory (and closed if requested) before
	6118	* this call returns.
	6119	*
	6120	* @param tree The tree to initialize.
	6121	* @param stdfile The FILE.
	6122	* @param max_bytes The maximum size of file to load, or 0 for unlimited size.
	6123	* @param close_when_done If true, fclose() will be called on the FILE when it
	6124	* is no longer needed. If false, the file will not be closed when
	6125	* reading is done.
	6126	*
	6127	* @warning The tree will read all data in the FILE before parsing it. If this
	6128	* is used on stdin, the parser will block until it is closed, even if
	6129	* a complete message has been written to it!
	6130	*/
	6131	void mpack_tree_init_stdfile(mpack_tree_t* tree, FILE* stdfile, size_t max_bytes, bool close_when_done);
	6132	#endif
	6133
	6134	/**
	6135	* @}
	6136	*/
	6137
	6138	/**
	6139	* @name Tree Functions
	6140	* @{
	6141	*/
	6142
	6143	/**
	6144	* Sets the maximum byte size and maximum number of nodes allowed per message.
	6145	*
	6146	* The default is SIZE_MAX (no limit) unless @ref mpack_tree_init_stream() is
	6147	* called (where maximums are required.)
	6148	*
	6149	* If a pool of nodes is used, the node limit is the lesser of this limit and
	6150	* the pool size.
	6151	*
	6152	* @param tree The tree parser
	6153	* @param max_message_size The maximum size of a message in bytes
	6154	* @param max_message_nodes The maximum number of nodes per message. See
	6155	* @ref mpack_node_data_t for the size of nodes.
	6156	*/
	6157	void mpack_tree_set_limits(mpack_tree_t* tree, size_t max_message_size,
	6158	size_t max_message_nodes);
	6159
	6160	/**
	6161	* Parses a MessagePack message into a tree of immutable nodes.
	6162	*
	6163	* If successful, the root node will be available under @ref mpack_tree_root().
	6164	* If not, an appropriate error will be flagged.
	6165	*
	6166	* This can be called repeatedly to parse a series of messages from a data
	6167	* source. When this is called, all previous nodes from this tree and their
	6168	* contents (including the root node) are invalidated.
	6169	*
	6170	* If this is called with a stream (see @ref mpack_tree_init_stream()), the
	6171	* stream must block until data is available. (Otherwise, if this is called on
	6172	* a non-blocking stream, parsing will fail with @ref mpack_error_io when the
	6173	* fill function returns 0.)
	6174	*
	6175	* There is no way to recover a tree in an error state. It must be destroyed.
	6176	*/
	6177	void mpack_tree_parse(mpack_tree_t* tree);
	6178
	6179	/**
	6180	* Attempts to parse a MessagePack message from a non-blocking stream into a
	6181	* tree of immutable nodes.
	6182	*
	6183	* A non-blocking read function must have been passed to the tree in
	6184	* mpack_tree_init_stream().
	6185	*
	6186	* If this returns true, a message is available under
	6187	* @ref mpack_tree_root(). The tree nodes and data will be valid until
	6188	* the next time a parse is started.
	6189	*
	6190	* If this returns false, no message is available, because either not enough
	6191	* data is available yet or an error has occurred. You must check the tree for
	6192	* errors whenever this returns false. If there is no error, you should try
	6193	* again later when more data is available. (You will want to select()/poll()
	6194	* on the underlying socket or use some other asynchronous mechanism to
	6195	* determine when it has data.)
	6196	*
	6197	* There is no way to recover a tree in an error state. It must be destroyed.
	6198	*
	6199	* @see mpack_tree_init_stream()
	6200	*/
	6201	bool mpack_tree_try_parse(mpack_tree_t* tree);
	6202
	6203	/**
	6204	* Returns the root node of the tree, if the tree is not in an error state.
	6205	* Returns a nil node otherwise.
	6206	*
	6207	* @warning You must call mpack_tree_parse() before calling this. If
	6208	* @ref mpack_tree_parse() was never called, the tree will assert.
	6209	*/
	6210	mpack_node_t mpack_tree_root(mpack_tree_t* tree);
	6211
	6212	/**
	6213	* Returns the error state of the tree.
	6214	*/
	6215	MPACK_INLINE mpack_error_t mpack_tree_error(mpack_tree_t* tree) {
	6216	return tree->error;
	6217	}
	6218
	6219	/**
	6220	* Returns the size in bytes of the current parsed message.
	6221	*
	6222	* If there is something in the buffer after the MessagePack object, this can
	6223	* be used to find it.
	6224	*
	6225	* This is zero if an error occurred during tree parsing (since the
	6226	* portion of the data that the first complete object occupies cannot
	6227	* be determined if the data is invalid or corrupted.)
	6228	*/
	6229	MPACK_INLINE size_t mpack_tree_size(mpack_tree_t* tree) {
	6230	return tree->size;
	6231	}
	6232
	6233	/**
	6234	* Destroys the tree.
	6235	*/
	6236	mpack_error_t mpack_tree_destroy(mpack_tree_t* tree);
	6237
	6238	/**
	6239	* Sets the custom pointer to pass to the tree callbacks, such as teardown.
	6240	*
	6241	* @param tree The MPack tree.
	6242	* @param context User data to pass to the tree callbacks.
	6243	*
	6244	* @see mpack_reader_context()
	6245	*/
	6246	MPACK_INLINE void mpack_tree_set_context(mpack_tree_t* tree, void* context) {
	6247	tree->context = context;
	6248	}
	6249
	6250	/**
	6251	* Returns the custom context for tree callbacks.
	6252	*
	6253	* @see mpack_tree_set_context
	6254	* @see mpack_tree_init_stream
	6255	*/
	6256	MPACK_INLINE void* mpack_tree_context(mpack_tree_t* tree) {
	6257	return tree->context;
	6258	}
	6259
	6260	/**
	6261	* Sets the error function to call when an error is flagged on the tree.
	6262	*
	6263	* This should normally be used with mpack_tree_set_context() to register
	6264	* a custom pointer to pass to the error function.
	6265	*
	6266	* See the definition of mpack_tree_error_t for more information about
	6267	* what you can do from an error callback.
	6268	*
	6269	* @see mpack_tree_error_t
	6270	* @param tree The MPack tree.
	6271	* @param error_fn The function to call when an error is flagged on the tree.
	6272	*/
	6273	MPACK_INLINE void mpack_tree_set_error_handler(mpack_tree_t* tree, mpack_tree_error_t error_fn) {
	6274	tree->error_fn = error_fn;
	6275	}
	6276
	6277	/**
	6278	* Sets the teardown function to call when the tree is destroyed.
	6279	*
	6280	* This should normally be used with mpack_tree_set_context() to register
	6281	* a custom pointer to pass to the teardown function.
	6282	*
	6283	* @param tree The MPack tree.
	6284	* @param teardown The function to call when the tree is destroyed.
	6285	*/
	6286	MPACK_INLINE void mpack_tree_set_teardown(mpack_tree_t* tree, mpack_tree_teardown_t teardown) {
	6287	tree->teardown = teardown;
	6288	}
	6289
	6290	/**
	6291	* Places the tree in the given error state, calling the error callback if one
	6292	* is set.
	6293	*
	6294	* This allows you to externally flag errors, for example if you are validating
	6295	* data as you read it.
	6296	*
	6297	* If the tree is already in an error state, this call is ignored and no
	6298	* error callback is called.
	6299	*/
	6300	void mpack_tree_flag_error(mpack_tree_t* tree, mpack_error_t error);
	6301
	6302	/**
	6303	* @}
	6304	*/
	6305
	6306	/**
	6307	* @name Node Core Functions
	6308	* @{
	6309	*/
	6310
	6311	/**
	6312	* Places the node's tree in the given error state, calling the error callback
	6313	* if one is set.
	6314	*
	6315	* This allows you to externally flag errors, for example if you are validating
	6316	* data as you read it.
	6317	*
	6318	* If the tree is already in an error state, this call is ignored and no
	6319	* error callback is called.
	6320	*/
	6321	void mpack_node_flag_error(mpack_node_t node, mpack_error_t error);
	6322
	6323	/**
	6324	* Returns the error state of the node's tree.
	6325	*/
	6326	MPACK_INLINE mpack_error_t mpack_node_error(mpack_node_t node) {
	6327	return mpack_tree_error(node.tree);
	6328	}
	6329
	6330	/**
	6331	* Returns a tag describing the given node, or a nil tag if the
	6332	* tree is in an error state.
	6333	*/
	6334	mpack_tag_t mpack_node_tag(mpack_node_t node);
	6335
	6336	/** @cond */
	6337
	6338	#if MPACK_DEBUG && MPACK_STDIO
	6339	/*
	6340	* Converts a node to a pseudo-JSON string for debugging purposes, placing the
	6341	* result in the given buffer with a null-terminator.
	6342	*
	6343	* If the buffer does not have enough space, the result will be truncated (but
	6344	* it is guaranteed to be null-terminated.)
	6345	*
	6346	* This is only available in debug mode, and only if stdio is available (since
	6347	* it uses snprintf().) It's strictly for debugging purposes.
	6348	*/
	6349	void mpack_node_print_to_buffer(mpack_node_t node, char* buffer, size_t buffer_size);
	6350
	6351	/*
	6352	* Converts a node to pseudo-JSON for debugging purposes, calling the given
	6353	* callback as many times as is necessary to output the character data.
	6354	*
	6355	* No null-terminator or trailing newline will be written.
	6356	*
	6357	* This is only available in debug mode, and only if stdio is available (since
	6358	* it uses snprintf().) It's strictly for debugging purposes.
	6359	*/
	6360	void mpack_node_print_to_callback(mpack_node_t node, mpack_print_callback_t callback, void* context);
	6361
	6362	/*
	6363	* Converts a node to pseudo-JSON for debugging purposes
	6364	* and pretty-prints it to the given file.
	6365	*
	6366	* This is only available in debug mode, and only if stdio is available (since
	6367	* it uses snprintf().) It's strictly for debugging purposes.
	6368	*/
	6369	void mpack_node_print_to_file(mpack_node_t node, FILE* file);
	6370
	6371	/*
	6372	* Converts a node to pseudo-JSON for debugging purposes
	6373	* and pretty-prints it to stdout.
	6374	*
	6375	* This is only available in debug mode, and only if stdio is available (since
	6376	* it uses snprintf().) It's strictly for debugging purposes.
	6377	*/
	6378	MPACK_INLINE void mpack_node_print_to_stdout(mpack_node_t node) {
	6379	mpack_node_print_to_file(node, stdout);
	6380	}
	6381
	6382	/*
	6383	* Deprecated.
	6384	*
	6385	* \deprecated Renamed to mpack_node_print_to_stdout().
	6386	*/
	6387	MPACK_INLINE void mpack_node_print(mpack_node_t node) {
	6388	mpack_node_print_to_stdout(node);
	6389	}
	6390	#endif
	6391
	6392	/** @endcond */
	6393
	6394	/**
	6395	* @}
	6396	*/
	6397
	6398	/**
	6399	* @name Node Primitive Value Functions
	6400	* @{
	6401	*/
	6402
	6403	/**
	6404	* Returns the type of the node.
	6405	*/
	6406	mpack_type_t mpack_node_type(mpack_node_t node);
	6407
	6408	/**
	6409	* Returns true if the given node is a nil node; false otherwise.
	6410	*
	6411	* To ensure that a node is nil and flag an error otherwise, use
	6412	* mpack_node_nil().
	6413	*/
	6414	bool mpack_node_is_nil(mpack_node_t node);
	6415
	6416	/**
	6417	* Returns true if the given node handle indicates a missing node; false otherwise.
	6418	*
	6419	* To ensure that a node is missing and flag an error otherwise, use
	6420	* mpack_node_missing().
	6421	*/
	6422	bool mpack_node_is_missing(mpack_node_t node);
	6423
	6424	/**
	6425	* Checks that the given node is of nil type, raising @ref mpack_error_type
	6426	* otherwise.
	6427	*
	6428	* Use mpack_node_is_nil() to return whether the node is nil.
	6429	*/
	6430	void mpack_node_nil(mpack_node_t node);
	6431
	6432	/**
	6433	* Checks that the given node indicates a missing node, raising @ref
	6434	* mpack_error_type otherwise.
	6435	*
	6436	* Use mpack_node_is_missing() to return whether the node is missing.
	6437	*/
	6438	void mpack_node_missing(mpack_node_t node);
	6439
	6440	/**
	6441	* Returns the bool value of the node. If this node is not of the correct
	6442	* type, false is returned and mpack_error_type is raised.
	6443	*/
	6444	bool mpack_node_bool(mpack_node_t node);
	6445
	6446	/**
	6447	* Checks if the given node is of bool type with value true, raising
	6448	* mpack_error_type otherwise.
	6449	*/
	6450	void mpack_node_true(mpack_node_t node);
	6451
	6452	/**
	6453	* Checks if the given node is of bool type with value false, raising
	6454	* mpack_error_type otherwise.
	6455	*/
	6456	void mpack_node_false(mpack_node_t node);
	6457
	6458	/**
	6459	* Returns the 8-bit unsigned value of the node. If this node is not
	6460	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6461	*/
	6462	uint8_t mpack_node_u8(mpack_node_t node);
	6463
	6464	/**
	6465	* Returns the 8-bit signed value of the node. If this node is not
	6466	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6467	*/
	6468	int8_t mpack_node_i8(mpack_node_t node);
	6469
	6470	/**
	6471	* Returns the 16-bit unsigned value of the node. If this node is not
	6472	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6473	*/
	6474	uint16_t mpack_node_u16(mpack_node_t node);
	6475
	6476	/**
	6477	* Returns the 16-bit signed value of the node. If this node is not
	6478	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6479	*/
	6480	int16_t mpack_node_i16(mpack_node_t node);
	6481
	6482	/**
	6483	* Returns the 32-bit unsigned value of the node. If this node is not
	6484	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6485	*/
	6486	uint32_t mpack_node_u32(mpack_node_t node);
	6487
	6488	/**
	6489	* Returns the 32-bit signed value of the node. If this node is not
	6490	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6491	*/
	6492	int32_t mpack_node_i32(mpack_node_t node);
	6493
	6494	/**
	6495	* Returns the 64-bit unsigned value of the node. If this node is not
	6496	* of a compatible type, @ref mpack_error_type is raised, and zero is returned.
	6497	*/
	6498	uint64_t mpack_node_u64(mpack_node_t node);
	6499
	6500	/**
	6501	* Returns the 64-bit signed value of the node. If this node is not
	6502	* of a compatible type, @ref mpack_error_type is raised and zero is returned.
	6503	*/
	6504	int64_t mpack_node_i64(mpack_node_t node);
	6505
	6506	/**
	6507	* Returns the unsigned int value of the node.
	6508	*
	6509	* Returns zero if an error occurs.
	6510	*
	6511	* @throws mpack_error_type If the node is not an integer type or does not fit in the range of an unsigned int
	6512	*/
	6513	unsigned int mpack_node_uint(mpack_node_t node);
	6514
	6515	/**
	6516	* Returns the int value of the node.
	6517	*
	6518	* Returns zero if an error occurs.
	6519	*
	6520	* @throws mpack_error_type If the node is not an integer type or does not fit in the range of an int
	6521	*/
	6522	int mpack_node_int(mpack_node_t node);
	6523
	6524	/**
	6525	* Returns the float value of the node. The underlying value can be an
	6526	* integer, float or double; the value is converted to a float.
	6527	*
	6528	* @note Reading a double or a large integer with this function can incur a
	6529	* loss of precision.
	6530	*
	6531	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	6532	*/
	6533	float mpack_node_float(mpack_node_t node);
	6534
	6535	/**
	6536	* Returns the double value of the node. The underlying value can be an
	6537	* integer, float or double; the value is converted to a double.
	6538	*
	6539	* @note Reading a very large integer with this function can incur a
	6540	* loss of precision.
	6541	*
	6542	* @throws mpack_error_type if the underlying value is not a float, double or integer.
	6543	*/
	6544	double mpack_node_double(mpack_node_t node);
	6545
	6546	/**
	6547	* Returns the float value of the node. The underlying value must be a float,
	6548	* not a double or an integer. This ensures no loss of precision can occur.
	6549	*
	6550	* @throws mpack_error_type if the underlying value is not a float.
	6551	*/
	6552	float mpack_node_float_strict(mpack_node_t node);
	6553
	6554	/**
	6555	* Returns the double value of the node. The underlying value must be a float
	6556	* or double, not an integer. This ensures no loss of precision can occur.
	6557	*
	6558	* @throws mpack_error_type if the underlying value is not a float or double.
	6559	*/
	6560	double mpack_node_double_strict(mpack_node_t node);
	6561
	6562	#if MPACK_EXTENSIONS
	6563	/**
	6564	* Returns a timestamp.
	6565	*
	6566	* @note This requires @ref MPACK_EXTENSIONS.
	6567	*
	6568	* @throws mpack_error_type if the underlying value is not a timestamp.
	6569	*/
	6570	mpack_timestamp_t mpack_node_timestamp(mpack_node_t node);
	6571
	6572	/**
	6573	* Returns a timestamp's (signed) seconds since 1970-01-01T00:00:00Z.
	6574	*
	6575	* @note This requires @ref MPACK_EXTENSIONS.
	6576	*
	6577	* @throws mpack_error_type if the underlying value is not a timestamp.
	6578	*/
	6579	int64_t mpack_node_timestamp_seconds(mpack_node_t node);
	6580
	6581	/**
	6582	* Returns a timestamp's additional nanoseconds.
	6583	*
	6584	* @note This requires @ref MPACK_EXTENSIONS.
	6585	*
	6586	* @return A nanosecond count between 0 and 999,999,999 inclusive.
	6587	* @throws mpack_error_type if the underlying value is not a timestamp.
	6588	*/
	6589	uint32_t mpack_node_timestamp_nanoseconds(mpack_node_t node);
	6590	#endif
	6591
	6592	/**
	6593	* @}
	6594	*/
	6595
	6596	/**
	6597	* @name Node String and Data Functions
	6598	* @{
	6599	*/
	6600
	6601	/**
	6602	* Checks that the given node contains a valid UTF-8 string.
	6603	*
	6604	* If the string is invalid, this flags an error, which would cause subsequent calls
	6605	* to mpack_node_str() to return NULL and mpack_node_strlen() to return zero. So you
	6606	* can check the node for error immediately after calling this, or you can call those
	6607	* functions to use the data anyway and check for errors later.
	6608	*
	6609	* @throws mpack_error_type If this node is not a string or does not contain valid UTF-8.
	6610	*
	6611	* @param node The string node to test
	6612	*
	6613	* @see mpack_node_str()
	6614	* @see mpack_node_strlen()
	6615	*/
	6616	void mpack_node_check_utf8(mpack_node_t node);
	6617
	6618	/**
	6619	* Checks that the given node contains a valid UTF-8 string with no NUL bytes.
	6620	*
	6621	* This does not check that the string has a null-terminator! It only checks whether
	6622	* the string could safely be represented as a C-string by appending a null-terminator.
	6623	* (If the string does already contain a null-terminator, this will flag an error.)
	6624	*
	6625	* This is performed automatically by other UTF-8 cstr helper functions. Only
	6626	* call this if you will do something else with the data directly, but you still
	6627	* want to ensure it will be valid as a UTF-8 C-string.
	6628	*
	6629	* @throws mpack_error_type If this node is not a string, does not contain valid UTF-8,
	6630	* or contains a NUL byte.
	6631	*
	6632	* @param node The string node to test
	6633	*
	6634	* @see mpack_node_str()
	6635	* @see mpack_node_strlen()
	6636	* @see mpack_node_copy_utf8_cstr()
	6637	* @see mpack_node_utf8_cstr_alloc()
	6638	*/
	6639	void mpack_node_check_utf8_cstr(mpack_node_t node);
	6640
	6641	#if MPACK_EXTENSIONS
	6642	/**
	6643	* Returns the extension type of the given ext node.
	6644	*
	6645	* This returns zero if the tree is in an error state.
	6646	*
	6647	* @note This requires @ref MPACK_EXTENSIONS.
	6648	*/
	6649	int8_t mpack_node_exttype(mpack_node_t node);
	6650	#endif
	6651
	6652	/**
	6653	* Returns the number of bytes in the given bin node.
	6654	*
	6655	* This returns zero if the tree is in an error state.
	6656	*
	6657	* If this node is not a bin, @ref mpack_error_type is raised and zero is returned.
	6658	*/
	6659	size_t mpack_node_bin_size(mpack_node_t node);
	6660
	6661	/**
	6662	* Returns the length of the given str, bin or ext node.
	6663	*
	6664	* This returns zero if the tree is in an error state.
	6665	*
	6666	* If this node is not a str, bin or map, @ref mpack_error_type is raised and zero
	6667	* is returned.
	6668	*/
	6669	uint32_t mpack_node_data_len(mpack_node_t node);
	6670
	6671	/**
	6672	* Returns the length in bytes of the given string node. This does not
	6673	* include any null-terminator.
	6674	*
	6675	* This returns zero if the tree is in an error state.
	6676	*
	6677	* If this node is not a str, @ref mpack_error_type is raised and zero is returned.
	6678	*/
	6679	size_t mpack_node_strlen(mpack_node_t node);
	6680
	6681	/**
	6682	* Returns a pointer to the data contained by this node, ensuring the node is a
	6683	* string.
	6684	*
	6685	* @warning Strings are not null-terminated! Use one of the cstr functions
	6686	* to get a null-terminated string.
	6687	*
	6688	* The pointer is valid as long as the data backing the tree is valid.
	6689	*
	6690	* If this node is not a string, @ref mpack_error_type is raised and @c NULL is returned.
	6691	*
	6692	* @see mpack_node_copy_cstr()
	6693	* @see mpack_node_cstr_alloc()
	6694	* @see mpack_node_utf8_cstr_alloc()
	6695	*/
	6696	const char* mpack_node_str(mpack_node_t node);
	6697
	6698	/**
	6699	* Returns a pointer to the data contained by this node.
	6700	*
	6701	* @note Strings are not null-terminated! Use one of the cstr functions
	6702	* to get a null-terminated string.
	6703	*
	6704	* The pointer is valid as long as the data backing the tree is valid.
	6705	*
	6706	* If this node is not of a str, bin or map, @ref mpack_error_type is raised, and
	6707	* @c NULL is returned.
	6708	*
	6709	* @see mpack_node_copy_cstr()
	6710	* @see mpack_node_cstr_alloc()
	6711	* @see mpack_node_utf8_cstr_alloc()
	6712	*/
	6713	const char* mpack_node_data(mpack_node_t node);
	6714
	6715	/**
	6716	* Returns a pointer to the data contained by this bin node.
	6717	*
	6718	* The pointer is valid as long as the data backing the tree is valid.
	6719	*
	6720	* If this node is not a bin, @ref mpack_error_type is raised and @c NULL is
	6721	* returned.
	6722	*/
	6723	const char* mpack_node_bin_data(mpack_node_t node);
	6724
	6725	/**
	6726	* Copies the bytes contained by this node into the given buffer, returning the
	6727	* number of bytes in the node.
	6728	*
	6729	* @throws mpack_error_type If this node is not a str, bin or ext type
	6730	* @throws mpack_error_too_big If the string does not fit in the given buffer
	6731	*
	6732	* @param node The string node from which to copy data
	6733	* @param buffer A buffer in which to copy the node's bytes
	6734	* @param bufsize The size of the given buffer
	6735	*
	6736	* @return The number of bytes in the node, or zero if an error occurs.
	6737	*/
	6738	size_t mpack_node_copy_data(mpack_node_t node, char* buffer, size_t bufsize);
	6739
	6740	/**
	6741	* Checks that the given node contains a valid UTF-8 string and copies the
	6742	* string into the given buffer, returning the number of bytes in the string.
	6743	*
	6744	* @throws mpack_error_type If this node is not a string
	6745	* @throws mpack_error_too_big If the string does not fit in the given buffer
	6746	*
	6747	* @param node The string node from which to copy data
	6748	* @param buffer A buffer in which to copy the node's bytes
	6749	* @param bufsize The size of the given buffer
	6750	*
	6751	* @return The number of bytes in the node, or zero if an error occurs.
	6752	*/
	6753	size_t mpack_node_copy_utf8(mpack_node_t node, char* buffer, size_t bufsize);
	6754
	6755	/**
	6756	* Checks that the given node contains a string with no NUL bytes, copies the string
	6757	* into the given buffer, and adds a null terminator.
	6758	*
	6759	* If this node is not of a string type, @ref mpack_error_type is raised. If the string
	6760	* does not fit, @ref mpack_error_data is raised.
	6761	*
	6762	* If any error occurs, the buffer will contain an empty null-terminated string.
	6763	*
	6764	* @param node The string node from which to copy data
	6765	* @param buffer A buffer in which to copy the node's string
	6766	* @param size The size of the given buffer
	6767	*/
	6768	void mpack_node_copy_cstr(mpack_node_t node, char* buffer, size_t size);
	6769
	6770	/**
	6771	* Checks that the given node contains a valid UTF-8 string with no NUL bytes,
	6772	* copies the string into the given buffer, and adds a null terminator.
	6773	*
	6774	* If this node is not of a string type, @ref mpack_error_type is raised. If the string
	6775	* does not fit, @ref mpack_error_data is raised.
	6776	*
	6777	* If any error occurs, the buffer will contain an empty null-terminated string.
	6778	*
	6779	* @param node The string node from which to copy data
	6780	* @param buffer A buffer in which to copy the node's string
	6781	* @param size The size of the given buffer
	6782	*/
	6783	void mpack_node_copy_utf8_cstr(mpack_node_t node, char* buffer, size_t size);
	6784
	6785	#ifdef MPACK_MALLOC
	6786	/**
	6787	* Allocates a new chunk of data using MPACK_MALLOC() with the bytes
	6788	* contained by this node.
	6789	*
	6790	* The allocated data must be freed with MPACK_FREE() (or simply free()
	6791	* if MPack's allocator hasn't been customized.)
	6792	*
	6793	* @throws mpack_error_type If this node is not a str, bin or ext type
	6794	* @throws mpack_error_too_big If the size of the data is larger than the
	6795	* given maximum size
	6796	* @throws mpack_error_memory If an allocation failure occurs
	6797	*
	6798	* @param node The node from which to allocate and copy data
	6799	* @param maxsize The maximum size to allocate
	6800	*
	6801	* @return The allocated data, or NULL if any error occurs.
	6802	*/
	6803	char* mpack_node_data_alloc(mpack_node_t node, size_t maxsize);
	6804
	6805	/**
	6806	* Allocates a new null-terminated string using MPACK_MALLOC() with the string
	6807	* contained by this node.
	6808	*
	6809	* The allocated string must be freed with MPACK_FREE() (or simply free()
	6810	* if MPack's allocator hasn't been customized.)
	6811	*
	6812	* @throws mpack_error_type If this node is not a string or contains NUL bytes
	6813	* @throws mpack_error_too_big If the size of the string plus null-terminator
	6814	* is larger than the given maximum size
	6815	* @throws mpack_error_memory If an allocation failure occurs
	6816	*
	6817	* @param node The node from which to allocate and copy string data
	6818	* @param maxsize The maximum size to allocate, including the null-terminator
	6819	*
	6820	* @return The allocated string, or NULL if any error occurs.
	6821	*/
	6822	char* mpack_node_cstr_alloc(mpack_node_t node, size_t maxsize);
	6823
	6824	/**
	6825	* Allocates a new null-terminated string using MPACK_MALLOC() with the UTF-8
	6826	* string contained by this node.
	6827	*
	6828	* The allocated string must be freed with MPACK_FREE() (or simply free()
	6829	* if MPack's allocator hasn't been customized.)
	6830	*
	6831	* @throws mpack_error_type If this node is not a string, is not valid UTF-8,
	6832	* or contains NUL bytes
	6833	* @throws mpack_error_too_big If the size of the string plus null-terminator
	6834	* is larger than the given maximum size
	6835	* @throws mpack_error_memory If an allocation failure occurs
	6836	*
	6837	* @param node The node from which to allocate and copy string data
	6838	* @param maxsize The maximum size to allocate, including the null-terminator
	6839	*
	6840	* @return The allocated string, or NULL if any error occurs.
	6841	*/
	6842	char* mpack_node_utf8_cstr_alloc(mpack_node_t node, size_t maxsize);
	6843	#endif
	6844
	6845	/**
	6846	* Searches the given string array for a string matching the given
	6847	* node and returns its index.
	6848	*
	6849	* If the node does not match any of the given strings,
	6850	* @ref mpack_error_type is flagged. Use mpack_node_enum_optional()
	6851	* if you want to allow values other than the given strings.
	6852	*
	6853	* If any error occurs or if the tree is in an error state, @a count
	6854	* is returned.
	6855	*
	6856	* This can be used to quickly parse a string into an enum when the
	6857	* enum values range from 0 to @a count-1. If the last value in the
	6858	* enum is a special "count" value, it can be passed as the count,
	6859	* and the return value can be cast directly to the enum type.
	6860	*
	6861	* @code{.c}
	6862	* typedef enum { APPLE , BANANA , ORANGE , COUNT} fruit_t;
	6863	* const char* fruits[] = {"apple", "banana", "orange"};
	6864	*
	6865	* fruit_t fruit = (fruit_t)mpack_node_enum(node, fruits, COUNT);
	6866	* @endcode
	6867	*
	6868	* @param node The node
	6869	* @param strings An array of expected strings of length count
	6870	* @param count The number of strings
	6871	* @return The index of the matched string, or @a count in case of error
	6872	*/
	6873	size_t mpack_node_enum(mpack_node_t node, const char* strings[], size_t count);
	6874
	6875	/**
	6876	* Searches the given string array for a string matching the given node,
	6877	* returning its index or @a count if no strings match.
	6878	*
	6879	* If the value is not a string, or it does not match any of the
	6880	* given strings, @a count is returned and no error is flagged.
	6881	*
	6882	* If any error occurs or if the tree is in an error state, @a count
	6883	* is returned.
	6884	*
	6885	* This can be used to quickly parse a string into an enum when the
	6886	* enum values range from 0 to @a count-1. If the last value in the
	6887	* enum is a special "count" value, it can be passed as the count,
	6888	* and the return value can be cast directly to the enum type.
	6889	*
	6890	* @code{.c}
	6891	* typedef enum { APPLE , BANANA , ORANGE , COUNT} fruit_t;
	6892	* const char* fruits[] = {"apple", "banana", "orange"};
	6893	*
	6894	* fruit_t fruit = (fruit_t)mpack_node_enum_optional(node, fruits, COUNT);
	6895	* @endcode
	6896	*
	6897	* @param node The node
	6898	* @param strings An array of expected strings of length count
	6899	* @param count The number of strings
	6900	* @return The index of the matched string, or @a count in case of error
	6901	*/
	6902	size_t mpack_node_enum_optional(mpack_node_t node, const char* strings[], size_t count);
	6903
	6904	/**
	6905	* @}
	6906	*/
	6907
	6908	/**
	6909	* @name Compound Node Functions
	6910	* @{
	6911	*/
	6912
	6913	/**
	6914	* Returns the length of the given array node. Raises mpack_error_type
	6915	* and returns 0 if the given node is not an array.
	6916	*/
	6917	size_t mpack_node_array_length(mpack_node_t node);
	6918
	6919	/**
	6920	* Returns the node in the given array at the given index. If the node
	6921	* is not an array, @ref mpack_error_type is raised and a nil node is returned.
	6922	* If the given index is out of bounds, @ref mpack_error_data is raised and
	6923	* a nil node is returned.
	6924	*/
	6925	mpack_node_t mpack_node_array_at(mpack_node_t node, size_t index);
	6926
	6927	/**
	6928	* Returns the number of key/value pairs in the given map node. Raises
	6929	* mpack_error_type and returns 0 if the given node is not a map.
	6930	*/
	6931	size_t mpack_node_map_count(mpack_node_t node);
	6932
	6933	/**
	6934	* Returns the key node in the given map at the given index.
	6935	*
	6936	* A nil node is returned in case of error.
	6937	*
	6938	* @throws mpack_error_type if the node is not a map
	6939	* @throws mpack_error_data if the given index is out of bounds
	6940	*/
	6941	mpack_node_t mpack_node_map_key_at(mpack_node_t node, size_t index);
	6942
	6943	/**
	6944	* Returns the value node in the given map at the given index.
	6945	*
	6946	* A nil node is returned in case of error.
	6947	*
	6948	* @throws mpack_error_type if the node is not a map
	6949	* @throws mpack_error_data if the given index is out of bounds
	6950	*/
	6951	mpack_node_t mpack_node_map_value_at(mpack_node_t node, size_t index);
	6952
	6953	/**
	6954	* Returns the value node in the given map for the given integer key.
	6955	*
	6956	* The key must exist within the map. Use mpack_node_map_int_optional() to
	6957	* check for optional keys.
	6958	*
	6959	* The key must be unique. An error is flagged if the node has multiple
	6960	* entries with the given key.
	6961	*
	6962	* @throws mpack_error_type If the node is not a map
	6963	* @throws mpack_error_data If the node does not contain exactly one entry with the given key
	6964	*
	6965	* @return The value node for the given key, or a nil node in case of error
	6966	*/
	6967	mpack_node_t mpack_node_map_int(mpack_node_t node, int64_t num);
	6968
	6969	/**
	6970	* Returns the value node in the given map for the given integer key, or a
	6971	* missing node if the map does not contain the given key.
	6972	*
	6973	* The key must be unique. An error is flagged if the node has multiple
	6974	* entries with the given key.
	6975	*
	6976	* @throws mpack_error_type If the node is not a map
	6977	* @throws mpack_error_data If the node contains more than one entry with the given key
	6978	*
	6979	* @return The value node for the given key, or a missing node if the key does
	6980	* not exist, or a nil node in case of error
	6981	*
	6982	* @see mpack_node_is_missing()
	6983	*/
	6984	mpack_node_t mpack_node_map_int_optional(mpack_node_t node, int64_t num);
	6985
	6986	/**
	6987	* Returns the value node in the given map for the given unsigned integer key.
	6988	*
	6989	* The key must exist within the map. Use mpack_node_map_uint_optional() to
	6990	* check for optional keys.
	6991	*
	6992	* The key must be unique. An error is flagged if the node has multiple
	6993	* entries with the given key.
	6994	*
	6995	* @throws mpack_error_type If the node is not a map
	6996	* @throws mpack_error_data If the node does not contain exactly one entry with the given key
	6997	*
	6998	* @return The value node for the given key, or a nil node in case of error
	6999	*/
	7000	mpack_node_t mpack_node_map_uint(mpack_node_t node, uint64_t num);
	7001
	7002	/**
	7003	* Returns the value node in the given map for the given unsigned integer
	7004	* key, or a nil node if the map does not contain the given key.
	7005	*
	7006	* The key must be unique. An error is flagged if the node has multiple
	7007	* entries with the given key.
	7008	*
	7009	* @throws mpack_error_type If the node is not a map
	7010	* @throws mpack_error_data If the node contains more than one entry with the given key
	7011	*
	7012	* @return The value node for the given key, or a missing node if the key does
	7013	* not exist, or a nil node in case of error
	7014	*
	7015	* @see mpack_node_is_missing()
	7016	*/
	7017	mpack_node_t mpack_node_map_uint_optional(mpack_node_t node, uint64_t num);
	7018
	7019	/**
	7020	* Returns the value node in the given map for the given string key.
	7021	*
	7022	* The key must exist within the map. Use mpack_node_map_str_optional() to
	7023	* check for optional keys.
	7024	*
	7025	* The key must be unique. An error is flagged if the node has multiple
	7026	* entries with the given key.
	7027	*
	7028	* @throws mpack_error_type If the node is not a map
	7029	* @throws mpack_error_data If the node does not contain exactly one entry with the given key
	7030	*
	7031	* @return The value node for the given key, or a nil node in case of error
	7032	*/
	7033	mpack_node_t mpack_node_map_str(mpack_node_t node, const char* str, size_t length);
	7034
	7035	/**
	7036	* Returns the value node in the given map for the given string key, or a nil
	7037	* node if the map does not contain the given key.
	7038	*
	7039	* The key must be unique. An error is flagged if the node has multiple
	7040	* entries with the given key.
	7041	*
	7042	* @throws mpack_error_type If the node is not a map
	7043	* @throws mpack_error_data If the node contains more than one entry with the given key
	7044	*
	7045	* @return The value node for the given key, or a missing node if the key does
	7046	* not exist, or a nil node in case of error
	7047	*
	7048	* @see mpack_node_is_missing()
	7049	*/
	7050	mpack_node_t mpack_node_map_str_optional(mpack_node_t node, const char* str, size_t length);
	7051
	7052	/**
	7053	* Returns the value node in the given map for the given null-terminated
	7054	* string key.
	7055	*
	7056	* The key must exist within the map. Use mpack_node_map_cstr_optional() to
	7057	* check for optional keys.
	7058	*
	7059	* The key must be unique. An error is flagged if the node has multiple
	7060	* entries with the given key.
	7061	*
	7062	* @throws mpack_error_type If the node is not a map
	7063	* @throws mpack_error_data If the node does not contain exactly one entry with the given key
	7064	*
	7065	* @return The value node for the given key, or a nil node in case of error
	7066	*/
	7067	mpack_node_t mpack_node_map_cstr(mpack_node_t node, const char* cstr);
	7068
	7069	/**
	7070	* Returns the value node in the given map for the given null-terminated
	7071	* string key, or a nil node if the map does not contain the given key.
	7072	*
	7073	* The key must be unique. An error is flagged if the node has multiple
	7074	* entries with the given key.
	7075	*
	7076	* @throws mpack_error_type If the node is not a map
	7077	* @throws mpack_error_data If the node contains more than one entry with the given key
	7078	*
	7079	* @return The value node for the given key, or a missing node if the key does
	7080	* not exist, or a nil node in case of error
	7081	*
	7082	* @see mpack_node_is_missing()
	7083	*/
	7084	mpack_node_t mpack_node_map_cstr_optional(mpack_node_t node, const char* cstr);
	7085
	7086	/**
	7087	* Returns true if the given node map contains exactly one entry with the
	7088	* given integer key.
	7089	*
	7090	* The key must be unique. An error is flagged if the node has multiple
	7091	* entries with the given key.
	7092	*
	7093	* @throws mpack_error_type If the node is not a map
	7094	* @throws mpack_error_data If the node contains more than one entry with the given key
	7095	*/
	7096	bool mpack_node_map_contains_int(mpack_node_t node, int64_t num);
	7097
	7098	/**
	7099	* Returns true if the given node map contains exactly one entry with the
	7100	* given unsigned integer key.
	7101	*
	7102	* The key must be unique. An error is flagged if the node has multiple
	7103	* entries with the given key.
	7104	*
	7105	* @throws mpack_error_type If the node is not a map
	7106	* @throws mpack_error_data If the node contains more than one entry with the given key
	7107	*/
	7108	bool mpack_node_map_contains_uint(mpack_node_t node, uint64_t num);
	7109
	7110	/**
	7111	* Returns true if the given node map contains exactly one entry with the
	7112	* given string key.
	7113	*
	7114	* The key must be unique. An error is flagged if the node has multiple
	7115	* entries with the given key.
	7116	*
	7117	* @throws mpack_error_type If the node is not a map
	7118	* @throws mpack_error_data If the node contains more than one entry with the given key
	7119	*/
	7120	bool mpack_node_map_contains_str(mpack_node_t node, const char* str, size_t length);
	7121
	7122	/**
	7123	* Returns true if the given node map contains exactly one entry with the
	7124	* given null-terminated string key.
	7125	*
	7126	* The key must be unique. An error is flagged if the node has multiple
	7127	* entries with the given key.
	7128	*
	7129	* @throws mpack_error_type If the node is not a map
	7130	* @throws mpack_error_data If the node contains more than one entry with the given key
	7131	*/
	7132	bool mpack_node_map_contains_cstr(mpack_node_t node, const char* cstr);
	7133
	7134	/**
	7135	* @}
	7136	*/
	7137
	7138	/**
	7139	* @}
	7140	*/
	7141
	7142	#endif
	7143
	7144	MPACK_HEADER_END
	7145
	7146	#endif
	7147
	7148
	7149	#endif
	7150

+901

-0

backend/lib/protocol.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <bs.h>
	4	#include <mpack.h>
	5	#include <protocol.h>
	6	#include <trace.h>
	7
	8	enum {
	9	OBS_RRNAME_IDX = 0,
	10	OBS_RRTYPE_IDX = 1,
	11	OBS_RDATA_IDX = 2,
	12	OBS_SENSOR_IDX = 3,
	13	OBS_COUNT_IDX = 4,
	14	OBS_FIRST_SEEN_IDX = 5,
	15	OBS_LAST_SEEN_IDX = 6,
	16	OBS_FIELDS = 7
	17	};
	18
	19	#define PROTOCOL_TYPED_MESSAGE_TYPE_KEY ("T")
	20	#define PROTOCOL_TYPED_MESSAGE_ENCODED_KEY ("M")
	21
	22	#define PROTOCOL_BACKUP_REQUEST_PATH_KEY ("P")
	23
	24	#define PROTOCOL_DUMP_REQUEST_PATH_KEY ("P")
	25
	26	#define PROTOCOL_QUERY_REQUEST_QRDATA_KEY ("Qrdata")
	27	#define PROTOCOL_QUERY_REQUEST_QRRNAME_KEY ("Qrrname")
	28	#define PROTOCOL_QUERY_REQUEST_QRRTYPE_KEY ("Qrrtype")
	29	#define PROTOCOL_QUERY_REQUEST_QSENSORID_KEY ("QsensorID")
	30	#define PROTOCOL_QUERY_REQUEST_HRDATA_KEY ("Hrdata")
	31	#define PROTOCOL_QUERY_REQUEST_HRRNAME_KEY ("Hrrname")
	32	#define PROTOCOL_QUERY_REQUEST_HRRTYPE_KEY ("Hrrtype")
	33	#define PROTOCOL_QUERY_REQUEST_HSENSORID_KEY ("HsensorID")
	34	#define PROTOCOL_QUERY_REQUEST_LIMIT_KEY ("Limit")
	35
	36	#define PROTOCOL_INPUT_REQUEST_OBSERVATION_KEY0 ('O')
	37
	38	#define PROTOCOL_PDNS_ENTRY_RRNAME_KEY0 ('N')
	39	#define PROTOCOL_PDNS_ENTRY_RRTYPE_KEY0 ('T')
	40	#define PROTOCOL_PDNS_ENTRY_RDATA_KEY0 ('D')
	41	#define PROTOCOL_PDNS_ENTRY_SENSORID_KEY0 ('I')
	42	#define PROTOCOL_PDNS_ENTRY_COUNT_KEY0 ('C')
	43	#define PROTOCOL_PDNS_ENTRY_FIRSTSEEN_KEY0 ('F')
	44	#define PROTOCOL_PDNS_ENTRY_LASTSEEN_KEY0 ('L')
	45
	46	#define PROTOCOL_PDNS_ENTRY_RRNAME_KEY ("N")
	47	#define PROTOCOL_PDNS_ENTRY_RRTYPE_KEY ("T")
	48	#define PROTOCOL_PDNS_ENTRY_RDATA_KEY ("D")
	49	#define PROTOCOL_PDNS_ENTRY_SENSORID_KEY ("I")
	50	#define PROTOCOL_PDNS_ENTRY_COUNT_KEY ("C")
	51	#define PROTOCOL_PDNS_ENTRY_FIRSTSEEN_KEY ("F")
	52	#define PROTOCOL_PDNS_ENTRY_LASTSEEN_KEY ("L")
	53
	54	#define PROTOCOL_SCRTCH_SZ (1024 * 10)
	55	#define PROTOCOL_SCRTCH_BUFFERS (10)
	56
	57	struct protocol_stream_t {
	58	mpack_tree_t tree;
	59	void* usr;
	60	ssize_t (read_cb)(void usr, char* p, size_t p_sz);
	61	char scrtch[PROTOCOL_SCRTCH_BUFFERS][PROTOCOL_SCRTCH_SZ];
	62	};
	63
	64	struct protocol_dump_stream_t {
	65	mpack_reader_t reader;
	66	unsigned char scrtch[PROTOCOL_SCRTCH_SZ];
	67	};
	68
	69	static ssize_t blb_protocol_encode_outer_request(
	70	int type, char* p, size_t p_sz, size_t used_inner) {
	71	ASSERT(used_inner < p_sz);
	72	mpack_writer_t __wr = {0}, *wr = &__wr;
	73
	74	// encode outer message
	75	mpack_writer_init(wr, p + used_inner, p_sz - used_inner);
	76	mpack_start_map(wr, 2);
	77	mpack_write_cstr(wr, PROTOCOL_TYPED_MESSAGE_TYPE_KEY);
	78	mpack_write_int(wr, type);
	79	mpack_write_cstr(wr, PROTOCOL_TYPED_MESSAGE_ENCODED_KEY);
	80	mpack_write_bin(wr, p, used_inner);
	81	mpack_finish_map(wr);
	82	mpack_error_t outer_err = mpack_writer_error(wr);
	83	if(outer_err != mpack_ok) {
	84	L(log_debug("encoding outer message failed `%d`", outer_err));
	85	mpack_writer_destroy(wr);
	86	return (-1);
	87	}
	88
	89	size_t used_outer = mpack_writer_buffer_used(wr);
	90	X(log_debug("encoded outer message size `%zu`", used_outer));
	91	mpack_writer_destroy(wr);
	92	ASSERT((used_inner + used_outer) < p_sz);
	93	memmove(p, p + used_inner, used_outer);
	94	return (used_outer);
	95	}
	96
	97	ssize_t blb_protocol_encode_dump_request(
	98	const protocol_dump_request_t* r, char* p, size_t p_sz) {
	99	mpack_writer_t __wr = {0}, *wr = &__wr;
	100
	101	// encode inner message
	102	mpack_writer_init(wr, p, p_sz);
	103	mpack_start_map(wr, 1);
	104	mpack_write_cstr(wr, PROTOCOL_DUMP_REQUEST_PATH_KEY);
	105	mpack_write_str(wr, r->path, r->path_len);
	106	mpack_finish_map(wr);
	107	mpack_error_t err = mpack_writer_error(wr);
	108	if(err != mpack_ok) {
	109	L(log_error("encoding inner msgpack data failed `%d`", err));
	110	mpack_writer_destroy(wr);
	111	return (-1);
	112	}
	113
	114	size_t used_inner = mpack_writer_buffer_used(wr);
	115	X(log_debug("encoded inner message size `%zu`", used_inner));
	116	ASSERT(used_inner < p_sz);
	117	mpack_writer_destroy(wr);
	118
	119	return (blb_protocol_encode_outer_request(
	120	PROTOCOL_DUMP_REQUEST, p, p_sz, used_inner));
	121	}
	122
	123	ssize_t blb_protocol_encode_backup_request(
	124	const protocol_backup_request_t* r, char* p, size_t p_sz) {
	125	mpack_writer_t __wr = {0}, *wr = &__wr;
	126
	127	// encode inner message
	128	mpack_writer_init(wr, p, p_sz);
	129	mpack_start_map(wr, 1);
	130	mpack_write_cstr(wr, PROTOCOL_BACKUP_REQUEST_PATH_KEY);
	131	mpack_write_str(wr, r->path, r->path_len);
	132	mpack_finish_map(wr);
	133	mpack_error_t err = mpack_writer_error(wr);
	134	if(err != mpack_ok) {
	135	L(log_error("encoding inner msgpack data failed `%d`", err));
	136	mpack_writer_destroy(wr);
	137	return (-1);
	138	}
	139
	140	size_t used_inner = mpack_writer_buffer_used(wr);
	141	X(log_debug("encoded inner message size `%zu`", used_inner));
	142	ASSERT(used_inner < p_sz);
	143	mpack_writer_destroy(wr);
	144
	145	return (blb_protocol_encode_outer_request(
	146	PROTOCOL_BACKUP_REQUEST, p, p_sz, used_inner));
	147	}
	148
	149	ssize_t blb_protocol_encode_dump_entry(
	150	const protocol_entry_t* entry, char* p, size_t p_sz) {
	151	mpack_writer_t __wr = {0}, *wr = &__wr;
	152	mpack_writer_init(wr, p, p_sz);
	153
	154	mpack_start_map(wr, OBS_FIELDS);
	155	mpack_write_uint(wr, OBS_RRNAME_IDX);
	156	mpack_write_bin(wr, entry->rrname, entry->rrname_len);
	157	mpack_write_uint(wr, OBS_RRTYPE_IDX);
	158	mpack_write_bin(wr, entry->rrtype, entry->rrtype_len);
	159	mpack_write_uint(wr, OBS_RDATA_IDX);
	160	mpack_write_bin(wr, entry->rdata, entry->rdata_len);
	161	mpack_write_uint(wr, OBS_SENSOR_IDX);
	162	mpack_write_bin(wr, entry->sensorid, entry->sensorid_len);
	163	mpack_write_uint(wr, OBS_COUNT_IDX);
	164	mpack_write_uint(wr, entry->count);
	165	mpack_write_uint(wr, OBS_FIRST_SEEN_IDX);
	166	mpack_write_uint(wr, entry->first_seen);
	167	mpack_write_uint(wr, OBS_LAST_SEEN_IDX);
	168	mpack_write_uint(wr, entry->last_seen);
	169	mpack_finish_map(wr);
	170
	171	mpack_error_t err = mpack_writer_error(wr);
	172	if(err != mpack_ok) {
	173	L(log_error("encoding dump entry failed with mpack_error_t `%d`", err));
	174	mpack_writer_destroy(wr);
	175	return (-1);
	176	}
	177
	178	size_t used = mpack_writer_buffer_used(wr);
	179	X(log_debug("encoded dump entry size `%zu`", used));
	180
	181	ASSERT(used < p_sz);
	182
	183	mpack_writer_destroy(wr);
	184
	185	return (used);
	186	}
	187
	188	ssize_t blb_protocol_encode_entry(
	189	const protocol_entry_t* entry, char* p, size_t p_sz) {
	190	mpack_writer_t __wr = {0}, *wr = &__wr;
	191	mpack_writer_init(wr, p, p_sz);
	192
	193	mpack_start_map(wr, 7);
	194	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_COUNT_KEY);
	195	mpack_write_uint(wr, entry->count);
	196	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_FIRSTSEEN_KEY);
	197	mpack_write_timestamp_seconds(wr, entry->first_seen);
	198	// mpack_write_uint( wr, entry->first_seen );
	199	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_LASTSEEN_KEY);
	200	mpack_write_timestamp_seconds(wr, entry->last_seen);
	201	// mpack_write_uint( wr, entry->last_seen );
	202	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_RDATA_KEY);
	203	mpack_write_str(wr, entry->rdata, entry->rdata_len);
	204	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_RRNAME_KEY);
	205	mpack_write_str(wr, entry->rrname, entry->rrname_len);
	206	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_RRTYPE_KEY);
	207	mpack_write_str(wr, entry->rrtype, entry->rrtype_len);
	208	mpack_write_cstr(wr, PROTOCOL_PDNS_ENTRY_SENSORID_KEY);
	209	mpack_write_str(wr, entry->sensorid, entry->sensorid_len);
	210	mpack_finish_map(wr);
	211
	212	size_t used_inner = mpack_writer_buffer_used(wr);
	213
	214	mpack_error_t err = mpack_writer_error(wr);
	215	if(err != mpack_ok) {
	216	L(log_error("encoding inpot failed with mpack_error_t `%d`", err));
	217	mpack_writer_destroy(wr);
	218	return (-1);
	219	}
	220
	221	mpack_writer_destroy(wr);
	222
	223	return (used_inner);
	224	}
	225
	226	static ssize_t blb_protocol_encode_query(
	227	const protocol_query_request_t* query, char* p, size_t p_sz) {
	228	mpack_writer_t __wr = {0}, *wr = &__wr;
	229	mpack_writer_init(wr, p, p_sz);
	230
	231	mpack_start_map(wr, 9);
	232
	233	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_LIMIT_KEY);
	234	mpack_write_uint(wr, query->limit);
	235
	236	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_QRRNAME_KEY);
	237	mpack_write_str(wr, query->qrrname, query->qrrname_len);
	238	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_HRRNAME_KEY);
	239	mpack_write_bool(wr, query->qrrname != NULL);
	240
	241	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_QRDATA_KEY);
	242	mpack_write_str(wr, query->qrdata, query->qrdata_len);
	243	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_HRDATA_KEY);
	244	mpack_write_bool(wr, query->qrdata != NULL);
	245
	246	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_QRRTYPE_KEY);
	247	mpack_write_str(wr, query->qrrtype, query->qrrtype_len);
	248	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_HRRTYPE_KEY);
	249	mpack_write_bool(wr, query->qrrtype != NULL);
	250
	251	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_QSENSORID_KEY);
	252	mpack_write_str(wr, query->qsensorid, query->qsensorid_len);
	253	mpack_write_cstr(wr, PROTOCOL_QUERY_REQUEST_HSENSORID_KEY);
	254	mpack_write_bool(wr, query->qsensorid != NULL);
	255
	256	mpack_finish_map(wr);
	257
	258	size_t used_inner = mpack_writer_buffer_used(wr);
	259
	260	mpack_error_t err = mpack_writer_error(wr);
	261	if(err != mpack_ok) {
	262	L(log_error("encoding input failed with mpack_error_t `%d`", err));
	263	mpack_writer_destroy(wr);
	264	return (-1);
	265	}
	266
	267	mpack_writer_destroy(wr);
	268
	269	return (used_inner);
	270	}
	271
	272	ssize_t blb_protocol_encode_query_request(
	273	const protocol_query_request_t* query, char* p, size_t p_sz) {
	274	ssize_t used_inner = blb_protocol_encode_query(query, p, p_sz);
	275	if(used_inner <= 0) {
	276	L(log_error("blb_protocol_encode_qurey() failed"));
	277	return (-1);
	278	}
	279	return (blb_protocol_encode_outer_request(
	280	PROTOCOL_QUERY_REQUEST, p, p_sz, used_inner));
	281	}
	282
	283	ssize_t blb_protocol_encode_input_request(
	284	const protocol_input_request_t* input, char* p, size_t p_sz) {
	285	ssize_t used_inner = blb_protocol_encode_entry(&input->entry, p, p_sz);
	286	if(used_inner <= 0) {
	287	L(log_error("blb_protocol_encode_entry() failed"));
	288	return (-1);
	289	}
	290	return (blb_protocol_encode_outer_request(
	291	PROTOCOL_INPUT_REQUEST, p, p_sz, used_inner));
	292	}
	293
	294	ssize_t blb_protocol_encode_stream_start_response(char* p, size_t p_sz) {
	295	return (blb_protocol_encode_outer_request(
	296	PROTOCOL_QUERY_STREAM_START_RESPONSE, p, p_sz, 0));
	297	}
	298
	299	ssize_t blb_protocol_encode_stream_end_response(char* p, size_t p_sz) {
	300	return (blb_protocol_encode_outer_request(
	301	PROTOCOL_QUERY_STREAM_END_RESPONSE, p, p_sz, 0));
	302	}
	303
	304	ssize_t blb_protocol_encode_stream_entry(
	305	const protocol_entry_t* entry, char* p, size_t p_sz) {
	306	ssize_t rc = blb_protocol_encode_entry(entry, p, p_sz);
	307	if(rc <= 0) { return (-1); }
	308
	309	return (blb_protocol_encode_outer_request(
	310	PROTOCOL_QUERY_STREAM_DATA_RESPONSE, p, p_sz, rc));
	311	}
	312
	313	// read-from-stream api
	314
	315	#define PROTOCOL_POLL_READ_TIMEOUT (60)
	316
	317	static size_t blb_protocol_stream_cb(mpack_tree_t* tree, char* p, size_t p_sz) {
	318	protocol_stream_t* s = mpack_tree_context(tree);
	319
	320	ssize_t rc = s->read_cb(s->usr, p, p_sz);
	321	if(rc < 0) {
	322	L(log_error("read() failed: `%s`", strerror(errno)));
	323	mpack_tree_flag_error(tree, mpack_error_io);
	324	} else if(rc == 0) {
	325	X(log_debug("read() eof"));
	326	mpack_tree_flag_error(tree, mpack_error_eof);
	327	}
	328	return (rc);
	329	}
	330
	331	protocol_stream_t* blb_protocol_stream_new(
	332	void* usr,
	333	ssize_t (read_cb)(void usr, char* p, size_t p_sz),
	334	size_t max_sz,
	335	size_t max_nodes) {
	336	protocol_stream_t* s = blb_new(protocol_stream_t);
	337	if(s == NULL) {
	338	L(log_error("blb_new() failed"));
	339	return (NULL);
	340	}
	341	s->read_cb = read_cb;
	342	s->usr = usr;
	343	mpack_tree_init_stream(
	344	&s->tree, blb_protocol_stream_cb, s, max_sz, max_nodes);
	345	return (s);
	346	}
	347
	348	void blb_protocol_stream_teardown(protocol_stream_t* stream) {
	349	if(stream == NULL) { return; }
	350	mpack_tree_destroy(&stream->tree);
	351	blb_free(stream);
	352	}
	353
	354	static int blb_protocol_decode_input(
	355	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	356	const char* p = mpack_node_bin_data(payload);
	357	size_t p_sz = mpack_node_bin_size(payload);
	358	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	359	if(p == NULL \|\| p_sz == 0) {
	360	L(log_error("invalid message"));
	361	return (-1);
	362	}
	363
	364	WHEN_X {
	365	theTrace_lock();
	366	for(size_t i = 0; i < p_sz; i++) {
	367	log_inject("%02x ", (int)(unsigned char)p[i]);
	368	}
	369	log_inject("\n");
	370	theTrace_release();
	371	}
	372
	373	mpack_reader_t __rd = {0}, *rd = &__rd;
	374	mpack_reader_init(rd, (char*)p, p_sz, p_sz);
	375
	376	uint32_t cnt = mpack_expect_map(rd);
	377	mpack_error_t map_ok = mpack_reader_error(rd);
	378	if(cnt != 7 \|\| map_ok != mpack_ok) {
	379	L(log_error(
	380	"invalid inner message: map with `7` elements expected got `%u`", cnt));
	381	goto decode_error;
	382	}
	383
	384	ASSERT(cnt < PROTOCOL_SCRTCH_BUFFERS);
	385
	386	protocol_input_request_t* i = &out->u.input;
	387	out->ty = PROTOCOL_INPUT_REQUEST;
	388	uint32_t w = 0;
	389	for(uint32_t j = 0; j < cnt; j++) {
	390	char key[1] = {'\0'};
	391	(void)mpack_expect_str_buf(rd, key, 1);
	392	switch(key[0]) {
	393	case PROTOCOL_PDNS_ENTRY_COUNT_KEY0: {
	394	X(log_debug("got input request count"));
	395	i->entry.count = mpack_expect_uint(rd);
	396	break;
	397	}
	398	case PROTOCOL_PDNS_ENTRY_FIRSTSEEN_KEY0: {
	399	X(log_debug("got input request first seen"));
	400	mpack_timestamp_t ts = mpack_expect_timestamp(rd);
	401	i->entry.first_seen = ts.seconds;
	402	break;
	403	}
	404	case PROTOCOL_PDNS_ENTRY_LASTSEEN_KEY0: {
	405	X(log_debug("got input request last seen"));
	406	mpack_timestamp_t ts = mpack_expect_timestamp(rd);
	407	i->entry.last_seen = ts.seconds;
	408	break;
	409	}
	410	case PROTOCOL_PDNS_ENTRY_RRNAME_KEY0: {
	411	X(log_debug("got input request rrname"));
	412	i->entry.rrname_len =
	413	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	414	i->entry.rrname = stream->scrtch[w];
	415	w++;
	416	break;
	417	}
	418	case PROTOCOL_PDNS_ENTRY_RRTYPE_KEY0: {
	419	X(log_debug("got input request rrtype"));
	420	i->entry.rrtype_len =
	421	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	422	i->entry.rrtype = stream->scrtch[w];
	423	w++;
	424	break;
	425	}
	426	case PROTOCOL_PDNS_ENTRY_RDATA_KEY0: {
	427	X(log_debug("got input request rdata"));
	428	i->entry.rdata_len =
	429	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	430	i->entry.rdata = stream->scrtch[w];
	431	w++;
	432	break;
	433	}
	434	case PROTOCOL_PDNS_ENTRY_SENSORID_KEY0: {
	435	X(log_debug("got input request sensorid"));
	436	i->entry.sensorid_len =
	437	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	438	i->entry.sensorid = stream->scrtch[w];
	439	w++;
	440	break;
	441	}
	442	default:
	443	L(log_error(
	444	"invalid inner message: invalid key: %02x",
	445	(int)(unsigned char)key[0]));
	446	return (-1);
	447	}
	448	}
	449
	450	mpack_done_map(rd);
	451	if(mpack_reader_error(rd) != mpack_ok) {
	452	L(log_error("invalid inner message; map decode failed"));
	453	goto decode_error;
	454	}
	455
	456	mpack_reader_destroy(rd);
	457	return (0);
	458
	459	decode_error:
	460	mpack_reader_destroy(rd);
	461	return (-1);
	462	}
	463
	464	static int blb_protocol_decode_query(
	465	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	466	const char* p = mpack_node_bin_data(payload);
	467	size_t p_sz = mpack_node_bin_size(payload);
	468	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	469	if(p == NULL \|\| p_sz == 0) {
	470	L(log_error("invalid message"));
	471	return (-1);
	472	}
	473
	474	WHEN_X {
	475	theTrace_lock();
	476	for(size_t i = 0; i < p_sz; i++) {
	477	log_inject("%02x ", (int)(unsigned char)p[i]);
	478	}
	479	log_inject("\n");
	480	theTrace_release();
	481	}
	482
	483	mpack_reader_t __rd = {0}, *rd = &__rd;
	484	mpack_reader_init(rd, (char*)p, p_sz, p_sz);
	485
	486	uint32_t cnt = mpack_expect_map(rd);
	487	if(cnt != 9 \|\| mpack_reader_error(rd) != mpack_ok) {
	488	L(log_error("invalid inner message: query map expected"));
	489	goto decode_error;
	490	}
	491
	492	ASSERT(cnt < PROTOCOL_SCRTCH_BUFFERS);
	493
	494	struct have_t {
	495	bool hrrname;
	496	bool hrdata;
	497	bool hrrtype;
	498	bool hsensorid;
	499	};
	500
	501	struct have_t __h = {0}, *h = &__h;
	502	protocol_query_request_t* q = &out->u.query;
	503	out->ty = PROTOCOL_QUERY_REQUEST;
	504	uint32_t w = 0;
	505	for(uint32_t j = 0; j < cnt; j++) {
	506	char key[64] = {'\0'};
	507	size_t key_len = mpack_expect_str_buf(rd, key, sizeof(key));
	508	if(key_len <= 0) {
	509	L(log_error("invalid inner message: invalid key"));
	510	goto decode_error;
	511	}
	512	if(strncmp(key, PROTOCOL_QUERY_REQUEST_LIMIT_KEY, key_len) == 0) {
	513	X(log_debug("got query request limit"));
	514	q->limit = mpack_expect_int(rd);
	515	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_QRRNAME_KEY, key_len) == 0) {
	516	X(log_debug("got input request rrname"));
	517	q->qrrname_len =
	518	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	519	q->qrrname = stream->scrtch[w];
	520	w++;
	521	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_HRRNAME_KEY, key_len) == 0) {
	522	X(log_debug("got input request have rrname"));
	523	h->hrrname = mpack_expect_bool(rd);
	524	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_QRRTYPE_KEY, key_len) == 0) {
	525	X(log_debug("got input request rrtype"));
	526	q->qrrtype_len =
	527	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	528	q->qrrtype = stream->scrtch[w];
	529	w++;
	530	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_HRRTYPE_KEY, key_len) == 0) {
	531	X(log_debug("got input request have rrtype"));
	532	h->hrrtype = mpack_expect_bool(rd);
	533	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_QRDATA_KEY, key_len) == 0) {
	534	X(log_debug("got input request rdata"));
	535	q->qrdata_len =
	536	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	537	q->qrdata = stream->scrtch[w];
	538	w++;
	539	} else if(strncmp(key, PROTOCOL_QUERY_REQUEST_HRDATA_KEY, key_len) == 0) {
	540	X(log_debug("got input request have rdata"));
	541	h->hrdata = mpack_expect_bool(rd);
	542	} else if(
	543	strncmp(key, PROTOCOL_QUERY_REQUEST_QSENSORID_KEY, key_len) == 0) {
	544	X(log_debug("got input request sensorid"));
	545	q->qsensorid_len =
	546	mpack_expect_str_buf(rd, stream->scrtch[w], PROTOCOL_SCRTCH_SZ);
	547	q->qsensorid = stream->scrtch[w];
	548	w++;
	549	} else if(
	550	strncmp(key, PROTOCOL_QUERY_REQUEST_HSENSORID_KEY, key_len) == 0) {
	551	X(log_debug("got input request have sensorid"));
	552	h->hsensorid = mpack_expect_bool(rd);
	553	} else {
	554	L(log_error(
	555	"invalid inner message: unkown key: `%.*s`", (int)key_len, key));
	556	goto decode_error;
	557	}
	558	}
	559	mpack_done_map(rd);
	560	if(mpack_reader_error(rd) != mpack_ok) {
	561	L(log_error("invalid inner message; decode query request failed"));
	562	goto decode_error;
	563	}
	564
	565	if(!h->hsensorid) { q->qsensorid_len = 0; }
	566	if(!h->hrrname) { q->qrrname_len = 0; }
	567	if(!h->hrrtype) { q->qrrtype_len = 0; }
	568	if(!h->hrdata) { q->qrdata_len = 0; }
	569
	570	mpack_reader_destroy(rd);
	571	return (0);
	572
	573	decode_error:
	574	mpack_reader_destroy(rd);
	575	return (-1);
	576	}
	577
	578	static int blb_protocol_decode_backup(
	579	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	580	const char* p = mpack_node_bin_data(payload);
	581	size_t p_sz = mpack_node_bin_size(payload);
	582	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	583	if(p == NULL \|\| p_sz == 0) {
	584	L(log_error("invalid message"));
	585	return (-1);
	586	}
	587
	588	WHEN_X {
	589	theTrace_lock();
	590	for(size_t i = 0; i < p_sz; i++) {
	591	log_inject("%02x ", (int)(unsigned char)p[i]);
	592	}
	593	log_inject("\n");
	594	theTrace_release();
	595	}
	596
	597	mpack_reader_t __rd = {0}, *rd = &__rd;
	598	mpack_reader_init(rd, (char*)p, p_sz, p_sz);
	599
	600	uint32_t cnt = mpack_expect_map(rd);
	601	if(cnt != 1 \|\| mpack_reader_error(rd) != mpack_ok) {
	602	L(log_error("invalid inner message: backup map expected"));
	603	goto decode_error;
	604	}
	605
	606	ASSERT(cnt < PROTOCOL_SCRTCH_BUFFERS);
	607
	608	char key[1] = {'\0'};
	609	(void)mpack_expect_str_buf(rd, key, 1);
	610	if(key[0] != PROTOCOL_BACKUP_REQUEST_PATH_KEY[0]) {
	611	L(log_error("invalid inner message: path key expected"));
	612	goto decode_error;
	613	}
	614
	615	protocol_backup_request_t* b = &out->u.backup;
	616	out->ty = PROTOCOL_BACKUP_REQUEST;
	617	b->path_len = mpack_expect_str_buf(rd, stream->scrtch[0], PROTOCOL_SCRTCH_SZ);
	618	b->path = stream->scrtch[0];
	619
	620	mpack_done_map(rd);
	621	if(mpack_reader_error(rd) != mpack_ok) {
	622	L(log_error("invalid inner message; decode backup request failed"));
	623	goto decode_error;
	624	}
	625
	626	mpack_reader_destroy(rd);
	627	return (0);
	628
	629	decode_error:
	630	mpack_reader_destroy(rd);
	631	return (-1);
	632	}
	633
	634	static int blb_protocol_decode_dump(
	635	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	636	const char* p = mpack_node_bin_data(payload);
	637	size_t p_sz = mpack_node_bin_size(payload);
	638	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	639	if(p == NULL \|\| p_sz == 0) {
	640	L(log_error("invalid message"));
	641	return (-1);
	642	}
	643
	644	WHEN_X {
	645	theTrace_lock();
	646	for(size_t i = 0; i < p_sz; i++) {
	647	log_inject("%02x ", (int)(unsigned char)p[i]);
	648	}
	649	log_inject("\n");
	650	theTrace_release();
	651	}
	652
	653	mpack_reader_t __rd = {0}, *rd = &__rd;
	654	mpack_reader_init(rd, (char*)p, p_sz, p_sz);
	655
	656	uint32_t cnt = mpack_expect_map(rd);
	657	if(cnt != 1 \|\| mpack_reader_error(rd) != mpack_ok) {
	658	L(log_error("invalid inner message: dump map expected"));
	659	goto decode_error;
	660	}
	661
	662	ASSERT(cnt < PROTOCOL_SCRTCH_BUFFERS);
	663
	664	char key[1] = {'\0'};
	665	(void)mpack_expect_str_buf(rd, key, 1);
	666	if(key[0] != PROTOCOL_DUMP_REQUEST_PATH_KEY[0]) {
	667	L(log_error("invalid inner message: path key expected"));
	668	goto decode_error;
	669	}
	670
	671	protocol_dump_request_t* d = &out->u.dump;
	672	out->ty = PROTOCOL_DUMP_REQUEST;
	673	d->path_len = mpack_expect_str_buf(rd, stream->scrtch[0], PROTOCOL_SCRTCH_SZ);
	674	d->path = stream->scrtch[0];
	675
	676	mpack_done_map(rd);
	677	if(mpack_reader_error(rd) != mpack_ok) {
	678	L(log_error("invalid inner message; decode dump request failed"));
	679	goto decode_error;
	680	}
	681
	682	mpack_reader_destroy(rd);
	683	return (0);
	684
	685	decode_error:
	686	mpack_reader_destroy(rd);
	687	return (-1);
	688	}
	689
	690	static int blb_protocol_decode_stream_start(
	691	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	692	const char* p = mpack_node_bin_data(payload);
	693	size_t p_sz = mpack_node_bin_size(payload);
	694	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	695	if(p == NULL \|\| p_sz != 0) {
	696	L(log_error("invalid message"));
	697	return (-1);
	698	}
	699	(void)stream;
	700	out->ty = PROTOCOL_QUERY_STREAM_START_RESPONSE;
	701	return (0);
	702	}
	703
	704	static int blb_protocol_decode_stream_end(
	705	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	706	const char* p = mpack_node_bin_data(payload);
	707	size_t p_sz = mpack_node_bin_size(payload);
	708	X(log_debug("encoded message ptr `%p` sz `%zu`", p, p_sz));
	709	if(p == NULL \|\| p_sz != 0) {
	710	L(log_error("invalid message"));
	711	return (-1);
	712	}
	713	(void)stream;
	714	out->ty = PROTOCOL_QUERY_STREAM_END_RESPONSE;
	715	return (0);
	716	}
	717
	718	static int blb_protocol_decode_stream_data(
	719	protocol_stream_t* stream, mpack_node_t payload, protocol_message_t* out) {
	720	int rc = blb_protocol_decode_input(stream, payload, out);
	721	if(rc != 0) { return (rc); }
	722	out->ty = PROTOCOL_QUERY_STREAM_DATA_RESPONSE;
	723	return (0);
	724	}
	725
	726	int blb_protocol_stream_decode(
	727	protocol_stream_t* stream, protocol_message_t* out) {
	728	mpack_tree_t* tree = &stream->tree;
	729	mpack_tree_parse(tree);
	730	mpack_error_t err = mpack_tree_error(tree);
	731	switch(err) {
	732	case mpack_ok: break;
	733	case mpack_error_eof: return (-1);
	734	default:
	735	L(log_error("mpack error `%s` `%d`", mpack_error_to_string(err), err));
	736	return (-2);
	737	}
	738
	739	mpack_node_t root = mpack_tree_root(tree);
	740	log_when(verbosity(3)) {
	741	log_enter();
	742	log_debug(
	743	"got message map entries `%zu` tree-size `%zu`",
	744	mpack_node_map_count(root),
	745	mpack_tree_size(tree));
	746	for(size_t i = 0; i < mpack_node_map_count(root); i++) {
	747	mpack_node_t key = mpack_node_map_key_at(root, i);
	748	log_debug("key[%zu] `%.*s`", i, 1, mpack_node_str(key));
	749	}
	750	log_leave();
	751	}
	752
	753	mpack_node_t type =
	754	mpack_node_map_cstr(root, PROTOCOL_TYPED_MESSAGE_TYPE_KEY);
	755	mpack_node_t payload =
	756	mpack_node_map_cstr(root, PROTOCOL_TYPED_MESSAGE_ENCODED_KEY);
	757	if(mpack_node_is_nil(type) \|\| mpack_node_is_nil(payload)) {
	758	L(log_error("invalid message received"));
	759	return (-1);
	760	}
	761	switch(mpack_node_int(type)) {
	762	case PROTOCOL_INPUT_REQUEST:
	763	X(log_debug("got input request"));
	764	return (blb_protocol_decode_input(stream, payload, out));
	765	case PROTOCOL_QUERY_REQUEST:
	766	X(log_debug("got query request"));
	767	return (blb_protocol_decode_query(stream, payload, out));
	768	case PROTOCOL_BACKUP_REQUEST:
	769	X(log_debug("got backup request"));
	770	return (blb_protocol_decode_backup(stream, payload, out));
	771	case PROTOCOL_DUMP_REQUEST:
	772	X(log_debug("got dump request"));
	773	return (blb_protocol_decode_dump(stream, payload, out));
	774	case PROTOCOL_QUERY_STREAM_START_RESPONSE:
	775	X(log_debug("got stream start response"));
	776	return (blb_protocol_decode_stream_start(stream, payload, out));
	777	case PROTOCOL_QUERY_STREAM_END_RESPONSE:
	778	X(log_debug("got stream end response"));
	779	return (blb_protocol_decode_stream_end(stream, payload, out));
	780	case PROTOCOL_QUERY_STREAM_DATA_RESPONSE:
	781	X(log_debug("got stream data response"));
	782	return (blb_protocol_decode_stream_data(stream, payload, out));
	783	default: L(log_error("invalid message type")); return (-1);
	784	}
	785	}
	786
	787	protocol_dump_stream_t* blb_protocol_dump_stream_new(FILE* file) {
	788	protocol_dump_stream_t* stream = blb_new(protocol_dump_stream_t);
	789	if(stream == NULL) { return (NULL); }
	790	mpack_reader_init_stdfile(&stream->reader, file, 0);
	791	return (stream);
	792	}
	793
	794	void blb_protocol_dump_stream_teardown(protocol_dump_stream_t* stream) {
	795	mpack_reader_destroy(&stream->reader);
	796	blb_free(stream);
	797	}
	798
	799	int blb_protocol_dump_stream_decode(
	800	protocol_dump_stream_t* stream, protocol_entry_t* entry) {
	801	mpack_reader_t* rd = &stream->reader;
	802	uint32_t cnt = mpack_expect_map(rd);
	803	mpack_error_t map_ok = mpack_reader_error(rd);
	804	if(map_ok != mpack_ok) {
	805	if(map_ok == mpack_error_eof) {
	806	V(log_debug("dump finished; eof reached"));
	807	return (-1);
	808	} else {
	809	L(log_error("mpack decode error `%d`", map_ok));
	810	return (-2);
	811	}
	812	}
	813	if(cnt != OBS_FIELDS) {
	814	L(log_error(
	815	"expected map with `%u` entries but got `%u` entries",
	816	OBS_FIELDS,
	817	cnt));
	818	return (-2);
	819	}
	820	bytestring_sink_t sink = bs_sink(stream->scrtch, PROTOCOL_SCRTCH_SZ);
	821	for(uint32_t i = 0; i < cnt; i++) {
	822	uint32_t field = mpack_expect_uint(rd);
	823	switch(field) {
	824	case OBS_RRTYPE_IDX: {
	825	size_t sz = mpack_expect_bin_buf(rd, (char*)sink.p, sink.available);
	826	entry->rrtype = (const char*)sink.p;
	827	entry->rrtype_len = sz;
	828	sink = bs_sink_slice0(&sink, sz);
	829	if(sink.p == 0) { return (-2); }
	830	break;
	831	}
	832	case OBS_RDATA_IDX: {
	833	size_t sz = mpack_expect_bin_buf(rd, (char*)sink.p, sink.available);
	834	entry->rdata = (const char*)sink.p;
	835	entry->rdata_len = sz;
	836	sink = bs_sink_slice0(&sink, sz);
	837	if(sink.p == 0) { return (-2); }
	838	break;
	839	}
	840	case OBS_SENSOR_IDX: {
	841	size_t sz = mpack_expect_bin_buf(rd, (char*)sink.p, sink.available);
	842	entry->sensorid = (const char*)sink.p;
	843	entry->sensorid_len = sz;
	844	sink = bs_sink_slice0(&sink, sz);
	845	if(sink.p == 0) { return (-2); }
	846	break;
	847	}
	848	case OBS_RRNAME_IDX: {
	849	size_t sz = mpack_expect_bin_buf(rd, (char*)sink.p, sink.available);
	850	entry->rrname = (const char*)sink.p;
	851	entry->rrname_len = sz;
	852	sink = bs_sink_slice0(&sink, sz);
	853	if(sink.p == 0) { return (-2); }
	854	break;
	855	}
	856	case OBS_COUNT_IDX: entry->count = mpack_expect_uint(rd); break;
	857	case OBS_LAST_SEEN_IDX: entry->last_seen = mpack_expect_uint(rd); break;
	858	case OBS_FIRST_SEEN_IDX: entry->first_seen = mpack_expect_uint(rd); break;
	859	default: L(log_error("unknown field index `%u`", field)); return (-1);
	860	}
	861	}
	862
	863	mpack_done_map(rd);
	864	mpack_error_t err = mpack_reader_error(rd);
	865	switch(err) {
	866	case mpack_ok: return (0);
	867	case mpack_error_eof: return (-1);
	868	default: return (-2);
	869	}
	870	}
	871
	872	void blb_protocol_log_entry(const protocol_entry_t* entry) {
	873	log_debug(
	874	"entry `%.s` `%.s` `%.s` `%.s` `%u` `%u` `%u`",
	875	(int)entry->rrname_len,
	876	entry->rrname,
	877	(int)entry->rrtype_len,
	878	entry->rrtype,
	879	(int)entry->rdata_len,
	880	entry->rdata,
	881	(int)entry->sensorid_len,
	882	entry->sensorid,
	883	entry->count,
	884	entry->first_seen,
	885	entry->last_seen);
	886	}
	887
	888	void blb_protocol_log_query(const protocol_query_request_t* q) {
	889	log_debug(
	890	"query `%.s` `%.s` `%.s` `%.s` `%u`",
	891	(int)q->qrrname_len,
	892	q->qrrname,
	893	(int)q->qrrtype_len,
	894	q->qrrtype,
	895	(int)q->qrdata_len,
	896	q->qrdata,
	897	(int)q->qsensorid_len,
	898	q->qsensorid,
	899	q->limit);
	900	}⏎

+122

-0

backend/lib/protocol.h less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#ifndef __PROTOCOL_H
	4	#define __PROTOCOL_H
	5
	6	#include <inttypes.h>
	7	#include <stdio.h>
	8	#include <stdlib.h>
	9
	10	#define PROTOCOL_INPUT_REQUEST 1
	11	#define PROTOCOL_QUERY_REQUEST 2
	12	#define PROTOCOL_BACKUP_REQUEST 3
	13	#define PROTOCOL_DUMP_REQUEST 4
	14	#define PROTOCOL_ERROR_RESPONSE 128
	15	#define PROTOCOL_QUERY_RESPONSE 129
	16	#define PROTOCOL_QUERY_STREAM_START_RESPONSE 130
	17	#define PROTOCOL_QUERY_STREAM_DATA_RESPONSE 131
	18	#define PROTOCOL_QUERY_STREAM_END_RESPONSE 132
	19
	20	typedef struct protocol_dump_request_t protocol_dump_request_t;
	21	struct protocol_dump_request_t {
	22	const char* path;
	23	size_t path_len;
	24	};
	25
	26	ssize_t blb_protocol_encode_dump_request(
	27	const protocol_dump_request_t* r, char* p, size_t p_sz);
	28
	29	typedef struct protocol_backup_request_t protocol_backup_request_t;
	30	struct protocol_backup_request_t {
	31	const char* path;
	32	size_t path_len;
	33	};
	34
	35	ssize_t blb_protocol_encode_backup_request(
	36	const protocol_backup_request_t* r, char* p, size_t p_sz);
	37
	38	typedef struct protocol_entry_t protocol_entry_t;
	39	struct protocol_entry_t {
	40	const char* rdata;
	41	size_t rdata_len;
	42	const char* rrname;
	43	size_t rrname_len;
	44	const char* rrtype;
	45	size_t rrtype_len;
	46	const char* sensorid;
	47	size_t sensorid_len;
	48	uint32_t count;
	49	uint32_t first_seen;
	50	uint32_t last_seen;
	51	};
	52
	53	typedef struct protocol_input_request_t protocol_input_request_t;
	54	struct protocol_input_request_t {
	55	protocol_entry_t entry;
	56	};
	57
	58	ssize_t blb_protocol_encode_input_request(
	59	const protocol_input_request_t* i, char* p, size_t p_sz);
	60	ssize_t blb_protocol_encode_entry(
	61	const protocol_entry_t* i, char* p, size_t p_sz);
	62
	63	typedef struct protocol_query_request_t protocol_query_request_t;
	64	struct protocol_query_request_t {
	65	int ty;
	66	const char* qrdata;
	67	size_t qrdata_len;
	68	const char* qrrname;
	69	size_t qrrname_len;
	70	const char* qrrtype;
	71	size_t qrrtype_len;
	72	const char* qsensorid;
	73	size_t qsensorid_len;
	74	int limit;
	75	};
	76
	77	ssize_t blb_protocol_encode_query_request(
	78	const protocol_query_request_t* q, char* p, size_t p_sz);
	79
	80	ssize_t blb_protocol_encode_stream_start_response(char* p, size_t p_sz);
	81	ssize_t blb_protocol_encode_stream_end_response(char* p, size_t p_sz);
	82	ssize_t blb_protocol_encode_stream_entry(
	83	const protocol_entry_t* entry, char* p, size_t p_sz);
	84	ssize_t blb_protocol_encode_dump_entry(
	85	const protocol_entry_t* entry, char* p, size_t p_sz);
	86
	87	typedef struct protocol_stream_t protocol_stream_t;
	88	protocol_stream_t* blb_protocol_stream_new(
	89	void* usr,
	90	ssize_t (read_cb)(void usr, char* p, size_t p_sz),
	91	size_t max_sz,
	92	size_t max_nodes);
	93
	94	typedef struct protocol_message_t protocol_message_t;
	95	struct protocol_message_t {
	96	int ty;
	97	union {
	98	protocol_input_request_t input;
	99	protocol_query_request_t query;
	100	protocol_backup_request_t backup;
	101	protocol_dump_request_t dump;
	102	protocol_entry_t entry;
	103	} u;
	104	};
	105	int blb_protocol_stream_decode(
	106	protocol_stream_t* stream, protocol_message_t* out);
	107
	108	void blb_protocol_stream_teardown(protocol_stream_t* stream);
	109
	110	typedef struct protocol_dump_stream_t protocol_dump_stream_t;
	111	protocol_dump_stream_t* blb_protocol_dump_stream_new(FILE* f);
	112
	113	void blb_protocol_dump_stream_teardown(protocol_dump_stream_t* stream);
	114
	115	int blb_protocol_dump_stream_decode(
	116	protocol_dump_stream_t* stream, protocol_entry_t* entry);
	117
	118	void blb_protocol_log_entry(const protocol_entry_t* entry);
	119	void blb_protocol_log_query(const protocol_query_request_t* q);
	120
	121	#endif⏎

+42

-0

backend/lib/rocksdb.nix less more

	0	{ stdenv, fetchFromGitHub, lib, bzip2, cmake, lz4, snappy, zstd, enableLite ? false }:
	1
	2	stdenv.mkDerivation rec {
	3	name = "rocksdb";
	4	version = "6.0.1";
	5
	6	src = fetchFromGitHub {
	7	owner = "facebook";
	8	repo = name;
	9	rev = "v${version}";
	10	sha256 = "02jmrvr1whc54q44n7qsah9vld2x40z00kxag8s721dgzcsk37sk";
	11	};
	12
	13	nativeBuildInputs = [ cmake ];
	14	buildInputs = [ bzip2 lz4 snappy zstd ];
	15
	16	cmakeFlags = [
	17	"-DCMAKE_BUILD_TYPE=Release"
	18	"-DPORTABLE=1"
	19	"-DWITH_JEMALLOC=0"
	20	"-DWITH_JNI=0"
	21	"-DWITH_TESTS=0"
	22	"-DWITH_TOOLS=0"
	23	"-DWITH_BZ2=1"
	24	"-DWITH_LZ4=1"
	25	"-DWITH_SNAPPY=1"
	26	"-DWITH_ZLIB=0"
	27	"-DWITH_ZSTD=1"
	28	(lib.optional
	29	(stdenv.hostPlatform.system == "i686-linux"
	30	\|\| stdenv.hostPlatform.system == "x86_64-linux")
	31	"-DFORCE_SSE42=1")
	32	(lib.optional enableLite "-DROCKSDB_LITE=1")
	33	];
	34
	35	meta = with stdenv.lib; {
	36	homepage = https://rocksdb.org;
	37	description = "A library that provides an embeddable, persistent key-value store for fast storage";
	38	license = licenses.bsd3;
	39	maintainers = with maintainers; [ adev magenbluten ];
	40	};
	41	}

+91

-0

backend/lib/trace.c less more

	0	// balboa
	1	// Copyright (c) 2018, 2019 DCSO GmbH
	2
	3	#include <inttypes.h>
	4	#include <pthread.h>
	5	#include <stdio.h>
	6	#include <sys/time.h>
	7	#include <time.h>
	8
	9	#include <trace.h>
	10
	11	// try to be https://tools.ietf.org/html/rfc5424 conform
	12
	13	static trace_t* theTrace = NULL;
	14
	15	void theTrace_set(trace_t* trace) {
	16	theTrace = trace;
	17	}
	18
	19	trace_t* theTrace_get(void) {
	20	return (theTrace);
	21	}
	22
	23	static void trace_lock(trace_t* trace) {
	24	(void)trace;
	25	pthread_mutex_lock(&trace->_lock);
	26	}
	27
	28	static void trace_release(trace_t* trace) {
	29	(void)trace;
	30	// fflush(stdout);
	31	pthread_mutex_unlock(&trace->_lock);
	32	}
	33
	34	static void trace_output_rfc5424(
	35	trace_t* trace, int priority, const char* fmt, va_list ap) {
	36	time_t now;
	37	time(&now);
	38	struct tm tm;
	39	localtime_r(&now, &tm);
	40	char b[128];
	41	size_t len = strftime(b, sizeof(b), "%FT%T%z", &tm);
	42	fprintf(
	43	trace->config.stream,
	44	"<%d>1 %.*s %s %s %d - - ",
	45	16 * 8 + priority,
	46	(int)len,
	47	b,
	48	trace->config.host,
	49	trace->config.app,
	50	trace->config.procid);
	51	vfprintf(trace->config.stream, fmt, ap);
	52	}
	53
	54	static void trace_output_systemd(
	55	trace_t* trace, int priority, const char* fmt, va_list ap) {
	56	fprintf(trace->config.stream, "<%d> ", priority);
	57	vfprintf(trace->config.stream, fmt, ap);
	58	}
	59
	60	static void trace_inject(trace_t* trace, const char* fmt, va_list ap) {
	61	vfprintf(trace->config.stream, fmt, ap);
	62	}
	63
	64	static void trace_flush(trace_t* trace) {
	65	(void)trace;
	66	fflush(stdout);
	67	}
	68
	69	static void trace_init(trace_t* trace, const trace_config_t* config) {
	70	trace->config = *config;
	71	atomic_store(&trace->verbosity, config->verbosity);
	72	if(config->rfc5424) { trace->output = trace_output_rfc5424; }
	73	pthread_mutex_init(&trace->_lock, NULL);
	74	}
	75
	76	static trace_t __theTrace_stdout = {.verbosity = ATOMIC_VAR_INIT(0),
	77	.config = {0},
	78	.release = trace_release,
	79	.init = trace_init,
	80	.lock = trace_lock,
	81	.output = trace_output_systemd,
	82	.inject = trace_inject,
	83	.flush = trace_flush};
	84
	85	static trace_t* const theTrace_stdout = &__theTrace_stdout;
	86
	87	void theTrace_stream_use(const trace_config_t* config) {
	88	theTrace_set(theTrace_stdout);
	89	theTrace_init(config);
	90	}

+162

-0

backend/lib/trace.h less more

	0	// balboa
	1	// Copyright (c) 2019 DCSO GmbH
	2
	3	#ifndef __TRACE_H
	4	#define __TRACE_H
	5
	6	#include <pthread.h>
	7	#include <stdarg.h>
	8	#include <stdatomic.h>
	9	#include <stdbool.h>
	10	#include <stdio.h>
	11	#include <stdlib.h>
	12
	13	#define V(body) \
	14	do { \
	15	if(theTrace_get_verbosity() > 0) { \
	16	theTrace_lock(); \
	17	body; \
	18	theTrace_release(); \
	19	} \
	20	} while(0)
	21	#define T(body) \
	22	do { \
	23	if(theTrace_get_verbosity() > 1) { \
	24	theTrace_lock(); \
	25	body; \
	26	theTrace_release(); \
	27	} \
	28	} while(0)
	29	#define X(body) \
	30	do { \
	31	if(theTrace_get_verbosity() > 2) { \
	32	theTrace_lock(); \
	33	body; \
	34	theTrace_release(); \
	35	} \
	36	} while(0)
	37	#define L(body) \
	38	do { \
	39	theTrace_lock(); \
	40	body; \
	41	theTrace_release(); \
	42	} while(0)
	43
	44	#define log_when(body) if(body)
	45	#define verbosity(lvl) (theTrace_get_verbosity() >= (lvl))
	46	#define log_enter() theTrace_lock()
	47	#define log_leave() theTrace_release()
	48
	49	// 0 Emergency: system is unusable
	50	// 1 Alert: action must be taken immediately
	51	// 2 Critical: critical conditions
	52	// 3 Error: error conditions
	53	// 4 Warning: warning conditions
	54	// 5 Notice: normal but significant condition
	55	// 6 Informational: informational messages
	56	// 7 Debug: debug-level messages
	57	#define log_emergency(fmt, ...) \
	58	do { \
	59	theTrace_output(0, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__); \
	60	exit(1); \
	61	} while(0)
	62	#define log_alert(fmt, ...) \
	63	theTrace_output(1, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	64	#define log_crit(fmt, ...) \
	65	theTrace_output(2, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	66	#define log_error(fmt, ...) \
	67	theTrace_output(3, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	68	#define log_warn(fmt, ...) \
	69	theTrace_output(4, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	70	#define log_notice(fmt, ...) \
	71	theTrace_output(5, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	72	#define log_info(fmt, ...) \
	73	theTrace_output(6, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	74	#define log_debug(fmt, ...) \
	75	theTrace_output(7, "(%s) " fmt "\n", __FUNCTION__, ##__VA_ARGS__)
	76	#define log_inject(fmt, ...) theTrace_inject(fmt, ##__VA_ARGS__)
	77
	78	#define WHEN_V if(theTrace_get_verbosity() > 0)
	79	#define WHEN_T if(theTrace_get_verbosity() > 1)
	80	#define WHEN_X if(theTrace_get_verbosity() > 2)
	81
	82	#define ASSERT(p) \
	83	do { \
	84	if(!(p)) { \
	85	log_emergency( \
	86	"assert failed `%s` (%s:%s:%d)", \
	87	#p, \
	88	__FILE__, \
	89	__FUNCTION__, \
	90	__LINE__); \
	91	} \
	92	} while(0)
	93
	94	typedef struct trace_config_t trace_config_t;
	95	struct trace_config_t {
	96	int verbosity;
	97	FILE* stream;
	98	const char* host;
	99	const char* app;
	100	pid_t procid;
	101	bool rfc5424;
	102	};
	103
	104	typedef struct trace_t trace_t;
	105	struct trace_t {
	106	atomic_int verbosity;
	107	pthread_mutex_t _lock;
	108	trace_config_t config;
	109	void (init)(trace_t trace, const trace_config_t* config);
	110	void (lock)(trace_t trace);
	111	void (release)(trace_t trace);
	112	void (output)(trace_t trace, int priority, const char* fmt, va_list ap);
	113	void (inject)(trace_t trace, const char* fmt, va_list ap);
	114	void (flush)(trace_t trace);
	115	};
	116
	117	void theTrace_set(trace_t* trace);
	118	trace_t* theTrace_get(void);
	119	void theTrace_stream_use(const trace_config_t* config);
	120
	121	__attribute__((format(printf, 2, 3))) static inline void theTrace_output(
	122	int priority, const char* fmt, ...) {
	123	va_list ap;
	124	va_start(ap, fmt);
	125	theTrace_get()->output(theTrace_get(), priority, fmt, ap);
	126	va_end(ap);
	127	}
	128
	129	__attribute__((format(printf, 1, 2))) static inline void theTrace_inject(
	130	const char* fmt, ...) {
	131	va_list ap;
	132	va_start(ap, fmt);
	133	theTrace_get()->inject(theTrace_get(), fmt, ap);
	134	va_end(ap);
	135	}
	136
	137	static inline void theTrace_lock(void) {
	138	theTrace_get()->lock(theTrace_get());
	139	}
	140
	141	static inline void theTrace_release(void) {
	142	theTrace_get()->release(theTrace_get());
	143	}
	144
	145	static inline int theTrace_get_verbosity(void) {
	146	return (atomic_load(&theTrace_get()->verbosity));
	147	}
	148
	149	static inline void theTrace_set_verbosity(int verbosity) {
	150	atomic_store(&theTrace_get()->verbosity, verbosity);
	151	}
	152
	153	static inline void theTrace_init(const trace_config_t* config) {
	154	theTrace_get()->init(theTrace_get(), config);
	155	}
	156
	157	static inline void theTrace_flush(void) {
	158	theTrace_get()->flush(theTrace_get());
	159	}
	160
	161	#endif

-0

backend/shell.nix less more

	0	{ pkgs ? import <nixpkgs> {}, unstable ? import <nixos-unstable> {} }: let
	1	rocksdb = pkgs.callPackage ./lib/rocksdb.nix {};
	2	in
	3	pkgs.mkShell {
	4	buildInputs = with pkgs; [ gcc gdb rocksdb unstable.clang_8 ];
	5	}

+18

-0

balboa-backend@.service less more

	0	[Unit]
	1	Description=Basic Little Book of Answers, '%i' Backend
	2	Documentation=https://github.com/DCSO/balboa
	3	After=network.target
	4	Before=balboa.service
	5
	6	[Service]
	7	SyslogIdentifier=balboa-%i
	8	EnvironmentFile=-/etc/default/balboa-%i
	9	ExecStart=/usr/bin/balboa-%i $BALBOA_BACKEND_ARGS
	10	ExecStop=/usr/bin/pkill balboa-%i
	11	PIDFile=/var/run/balboa/balboa-%i.pid
	12	LimitNOFILE=200000
	13	Restart=on-failure
	14	RestartSec=5
	15
	16	[Install]
	17	WantedBy=multi-user.target balboa.service

-1

balboa.service less more

8	8	ExecStart=/usr/bin/balboa serve $BALBOA_ARGS
9	9	ExecStop=/usr/bin/pkill balboa
10	10	PIDFile=/var/run/balboa/balboa.pid
11		LimitNOFILE=200000
12	11	Restart=on-failure
13	12	RestartSec=5
14	13

+44

-0

cmd/balboa/cmds/mock.go less more

	0	// balboa
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	package cmds
	4
	5	import (
	6	db "github.com/DCSO/balboa/backend/go"
	7	log "github.com/sirupsen/logrus"
	8	"github.com/spf13/cobra"
	9	)
	10
	11	// mockCmd represents the `mock` command
	12	var mockCmd = &cobra.Command{
	13	Use: "mock",
	14	Short: "Mock backend",
	15	Long: "Mock backend for debugging purposes",
	16	Run: func(cmd *cobra.Command, args []string) {
	17	log.Infof("running mock backend")
	18	var err error
	19	var verbose bool
	20	verbose, err = cmd.Flags().GetBool("verbose")
	21	if err != nil {
	22	log.Fatal(err)
	23	}
	24	if verbose {
	25	log.SetLevel(log.DebugLevel)
	26	}
	27
	28	var host string
	29	host, err = cmd.Flags().GetString("host")
	30	if err != nil {
	31	log.Fatal(err)
	32	}
	33
	34	db.Serve(host, nil)
	35	},
	36	}
	37
	38	func init() {
	39	rootCmd.AddCommand(mockCmd)
	40
	41	mockCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
	42	mockCmd.Flags().StringP("host", "H", "localhost:4242", "listen host and port of the backend")
	43	}

-1

cmd/balboa/cmds/query.go less more

58	58	var queryCmd = &cobra.Command{
59	59	Use: "query [netmask]",
60	60	Short: "Obtain information from pDNS about IP ranges",
61		Long: `This command allows to query a balboa endpoint for information regarding
	61	Long: `This command allows the user to query a balboa endpoint for information regarding
62	62	IPs from a given range.`,
63	63	Run: func(cmd *cobra.Command, args []string) {
64	64	var err error

-45

~~cmd/balboa/cmds/rocksdump.go~~ less more

0		// balboa
1		// Copyright (c) 2018, DCSO GmbH
2
3		package cmds
4
5		import (
6		"github.com/DCSO/balboa/db"
7		log "github.com/sirupsen/logrus"
8		"github.com/spf13/cobra"
9		)
10
11		var rocksdumpCmd = &cobra.Command{
12		Use: "rocksdump [netmask]",
13		Short: "Dump information from RocksDB",
14		Long: `This command allows directly dump bulk information from RocksDB.`,
15		Run: func(cmd *cobra.Command, args []string) {
16		var err error
17
18		if len(args) == 0 {
19		log.Fatal("needs database path as argument")
20		}
21
22		var verbose bool
23		verbose, err = cmd.Flags().GetBool("verbose")
24		if err != nil {
25		log.Fatal(err)
26		}
27		if verbose {
28		log.SetLevel(log.DebugLevel)
29		}
30
31		rdb, err := db.MakeRocksDBReadonly(args[0])
32		if err != nil {
33		log.Fatal(err)
34		}
35
36		rdb.Dump()
37		},
38		}
39
40		func init() {
41		rootCmd.AddCommand(rocksdumpCmd)
42
43		rocksdumpCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
44		}

+26

-22

cmd/balboa/cmds/serve.go less more

7	7	"os"
8	8	"os/signal"
9	9	"syscall"
	10	"time"
10	11
11	12	"github.com/DCSO/balboa/db"
12	13	"github.com/DCSO/balboa/feeder"

18	19
19	20	var serveCmd = &cobra.Command{
20	21	Use: "serve",
21		Short: "Run the balboa server",
22		Long: `This command starts the balboa server, accepting submissions and
	22	Short: "Run the balboa frontend",
	23	Long: `This command starts the balboa frontend, accepting submissions and
23	24	answering queries.`,
24	25	Run: func(cmd *cobra.Command, args []string) {
25	26	var err error

62	63	}
63	64	}
64	65
65		// Set up database from config file
66		var dbFile string
67		dbFile, err = cmd.Flags().GetString("dbconfig")
	66	// connect to backend
	67	host, err := cmd.Flags().GetString("host")
68	68	if err != nil {
69	69	log.Fatal(err)
70	70	}
71		cfgYaml, err := ioutil.ReadFile(dbFile)
72		if err != nil {
73		log.Fatal(err)
74		}
75		dbsetup, err := db.LoadSetup(cfgYaml)
76		if err != nil {
77		log.Fatal(err)
78		}
79		db.ObservationDB, err = dbsetup.Run()
	71	db.ObservationDB, err = db.MakeRemoteBackend(host, true)
80	72	if err != nil {
81	73	log.Fatal(err)
82	74	}

87	79	if err != nil {
88	80	log.Fatal(err)
89	81	}
90		cfgYaml, err = ioutil.ReadFile(feedersFile)
	82	cfgYaml, err := ioutil.ReadFile(feedersFile)
91	83	if err != nil {
92	84	log.Fatal(err)
93	85	}

101	93	}
102	94
103	95	// Start processing submissions
104		go db.ObservationDB.ConsumeFeed(observation.InChan)
	96	consumeDone := make(chan bool, 1)
	97	go func() {
	98	for {
	99	select {
	100	case <-consumeDone:
	101	log.Infof("ConsumeFeed() done")
	102	return
	103	default:
	104	log.Infof("ConsumeFeed() starting")
	105	db.ObservationDB.ConsumeFeed(observation.InChan)
	106	log.Info("ConsumeFeed() finished")
	107	time.Sleep(10 * time.Second)
	108	}
	109	}
	110	}()
105	111
106	112	// start query server
107	113	var port int

111	117	}
112	118	gql := query.GraphQLFrontend{}
113	119	gql.Run(int(port))
114
115	120	sigChan := make(chan os.Signal, 1)
116	121	done := make(chan bool, 1)
117	122	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
118	123	go func() {
119	124	sig := <-sigChan
120	125	log.Infof("received '%v' signal, shutting down", sig)
	126	close(consumeDone)
121	127	stopChan := make(chan bool)
122	128	fsetup.Stop(stopChan)
123	129	<-stopChan
124		stopChan = make(chan bool)
125		dbsetup.Stop(stopChan)
126		<-stopChan
	130	db.ObservationDB.Shutdown()
127	131	gql.Stop()
128	132	close(done)
129	133	}()

135	139	rootCmd.AddCommand(serveCmd)
136	140
137	141	serveCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
138		serveCmd.Flags().StringP("dbconfig", "d", "database.yaml", "database configuration file")
139		serveCmd.Flags().StringP("feeders", "f", "feeders.yaml", "feeders configuraion file")
	142	serveCmd.Flags().StringP("feeders", "f", "feeders.yaml", "feeders configuration file")
140	143	serveCmd.Flags().IntP("port", "p", 8080, "port for GraphQL server")
141	144	serveCmd.Flags().StringP("logfile", "l", "/var/log/balboa.log", "log file path")
142	145	serveCmd.Flags().BoolP("logjson", "j", true, "output log file as JSON")
	146	serveCmd.Flags().StringP("host", "H", "127.0.0.1:4242", "remote database host and port")
143	147	}

-9

~~database.yaml~~ less more

0		database:
1		name: Local RocksDB
2		type: rocksdb
3		db_path: /tmp/balboa
4
5		#database:
6		# name: Local Cassandra
7		# type: cassandra
8		# hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]

-2

db/db.go less more

9	9
10	10	// DB abstracts a database backend for observation storage.
11	11	type DB interface {
12		AddObservation(observation.InputObservation) observation.Observation
	12	//AddObservation(observation.InputObservation) observation.Observation
13	13	ConsumeFeed(chan observation.InputObservation)
	14	Backup(path string)
	15	Dump(path string)
14	16	Shutdown()
15	17	TotalCount() (int, error)
16		Search(string, string, string, string) ([]observation.Observation, error)
	18	Search(string, string, string, string, int) ([]observation.Observation, error)
17	19	}

-255

~~db/db_cassandra.go~~ less more

0		// balboa
1		// Copyright (c) 2018, DCSO GmbH
2
3		package db
4
5		import (
6		"time"
7
8		"github.com/DCSO/balboa/observation"
9
10		"github.com/gocql/gocql"
11		log "github.com/sirupsen/logrus"
12		)
13
14		// CassandraDB is a DB implementation based on Apache Cassandra.
15		type CassandraDB struct {
16		Cluster *gocql.ClusterConfig
17		Session *gocql.Session
18		StopChan chan bool
19		Nworkers uint
20		}
21
22		// MakeCassandraDB returns a new CassandraDB instance connecting to the
23		// provided hosts.
24		func MakeCassandraDB(hosts []string, username, password string, nofWorkers uint) (*CassandraDB, error) {
25		cluster := gocql.NewCluster(hosts...)
26		cluster.Keyspace = "balboa"
27		cluster.ProtoVersion = 4
28		cluster.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy())
29		cluster.RetryPolicy = &gocql.ExponentialBackoffRetryPolicy{NumRetries: 5}
30		cluster.Consistency = gocql.Two
31		if len(username) > 0 && len(password) > 0 {
32		cluster.Authenticator = gocql.PasswordAuthenticator{
33		Username: username,
34		Password: password,
35		}
36		}
37		gsession, err := cluster.CreateSession()
38		if err != nil {
39		return nil, err
40		}
41		//session := gockle.NewSession(gsession)
42		db := &CassandraDB{
43		Cluster: cluster,
44		Session: gsession,
45		StopChan: make(chan bool),
46		Nworkers: nofWorkers,
47		}
48		return db, nil
49		}
50
51		//func makeCassandraDBMock() (CassandraDB, gockle.SessionMock) {
52		// mock := &gockle.SessionMock{}
53		// db := &CassandraDB{
54		// //Session: mock,
55		// StopChan: make(chan bool),
56		// }
57		// return db, mock
58		//}
59
60		// AddObservation adds a single observation synchronously to the database.
61		func (db *CassandraDB) AddObservation(obs observation.InputObservation) observation.Observation {
62		//Not implemented
63		log.Warn("AddObservation() not yet implemented on Cassandra backend")
64		return observation.Observation{}
65		}
66
67		func (db *CassandraDB) runChunkWorker(inChan chan observation.InputObservation) {
68		rdataUpd := db.Session.Query(`UPDATE observations_by_rdata SET last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;`)
69		rrnameUpd := db.Session.Query(`UPDATE observations_by_rrname SET last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;`)
70		firstseenUpd := db.Session.Query(`INSERT INTO observations_firstseen (first_seen, rrname, rdata, rrtype, sensor_id) values (?, ?, ?, ?, ?) IF NOT EXISTS;`)
71		countsUpd := db.Session.Query(`UPDATE observations_counts SET count = count + ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;`)
72		for obs := range inChan {
73		select {
74		case <-db.StopChan:
75		log.Info("database ingest terminated")
76		return
77		default:
78		if obs.Rdata == "" {
79		obs.Rdata = "-"
80		}
81		if err := rdataUpd.Bind(obs.TimestampEnd, obs.Rdata, obs.Rrname, obs.Rrtype, obs.SensorID).Exec(); err != nil {
82		log.Error(err)
83		continue
84		}
85		if err := rrnameUpd.Bind(obs.TimestampEnd, obs.Rrname, obs.Rdata, obs.Rrtype, obs.SensorID).Exec(); err != nil {
86		log.Error(err)
87		continue
88		}
89		if err := firstseenUpd.Bind(obs.TimestampStart, obs.Rrname, obs.Rdata, obs.Rrtype, obs.SensorID).Exec(); err != nil {
90		log.Error(err)
91		continue
92		}
93		if err := countsUpd.Bind(obs.Count, obs.Rdata, obs.Rrname, obs.Rrtype, obs.SensorID).Exec(); err != nil {
94		log.Error(err)
95		continue
96		}
97		}
98		}
99		}
100
101		// ConsumeFeed accepts observations from a channel and queues them for
102		// database insertion.
103		func (db *CassandraDB) ConsumeFeed(inChan chan observation.InputObservation) {
104		var w uint
105		sendChan := make(chan observation.InputObservation, 500)
106		log.Debugf("Firing up %d workers", db.Nworkers)
107		for w = 1; w <= db.Nworkers; w++ {
108		go db.runChunkWorker(sendChan)
109		}
110		for {
111		select {
112		case <-db.StopChan:
113		log.Info("database ingest terminated")
114		return
115		case obs := <-inChan:
116		sendChan <- obs
117		}
118		}
119		}
120
121		// Search returns a slice of observations matching one or more criteria such
122		// as rdata, rrname, rrtype or sensor ID.
123		func (db CassandraDB) Search(qrdata, qrrname, qrrtype, qsensorID string) ([]observation.Observation, error) {
124		outs := make([]observation.Observation, 0)
125		var getQueryString string
126		var rdataFirst, hasSecond bool
127		var getQuery *gocql.Query
128
129		// determine appropriate table and parameterisation for query
130		if qrdata != nil {
131		rdataFirst = true
132		if qrrname != nil {
133		hasSecond = true
134		getQueryString = "SELECT * FROM observations_by_rdata WHERE rdata = ? and rrtype = ?"
135		} else {
136		hasSecond = false
137		getQueryString = "SELECT * FROM observations_by_rdata WHERE rdata = ?"
138		}
139		} else {
140		rdataFirst = false
141		if qrdata != nil {
142		hasSecond = true
143		getQueryString = "SELECT * FROM observations_by_rrname WHERE rrname = ? and rdata = ?"
144		} else {
145		hasSecond = false
146		getQueryString = "SELECT * FROM observations_by_rrname WHERE rrname = ?"
147		}
148		}
149		log.Debug(getQueryString)
150		getQuery = db.Session.Query(getQueryString)
151		getQuery.Consistency(gocql.One)
152
153		// do parameterised search
154		if rdataFirst {
155		if hasSecond {
156		getQuery.Bind(qrdata, qrrname)
157		} else {
158		getQuery.Bind(*qrdata)
159		}
160		} else {
161		if hasSecond {
162		getQuery.Bind(qrrname, qrdata)
163		} else {
164		getQuery.Bind(*qrrname)
165		}
166		}
167
168		// retrieve hits for initial queries
169		iter := getQuery.Iter()
170		for {
171		row := make(map[string]interface{})
172		if !iter.MapScan(row) {
173		break
174		}
175
176		if rrnameV, ok := row["rrname"]; ok {
177		var rdata, rrname, rrtype, sensorID string
178		var lastSeen int
179		rrname = rrnameV.(string)
180		if rdataV, ok := row["rdata"]; ok {
181		rdata = rdataV.(string)
182
183		// secondary filtering by sensor ID and RRType
184		if sensorIDV, ok := row["sensor_id"]; ok {
185		sensorID = sensorIDV.(string)
186		if qsensorID != nil && *qsensorID != sensorID {
187		continue
188		}
189		}
190		if rrtypeV, ok := row["rrtype"]; ok {
191		rrtype = rrtypeV.(string)
192		if qrrtype != nil && *qrrtype != rrtype {
193		continue
194		}
195		}
196		if lastSeenV, ok := row["last_seen"]; ok {
197		lastSeen = int(lastSeenV.(time.Time).Unix())
198		}
199
200		// we now have a result item
201		out := observation.Observation{
202		RRName: rrname,
203		RData: rdata,
204		RRType: rrtype,
205		SensorID: sensorID,
206		LastSeen: lastSeen,
207		}
208
209		// manual joins -> get additional data from counts table
210		tmpMap := make(map[string]interface{})
211		getCounters := db.Session.Query(`SELECT count FROM observations_counts WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?`).Bind(rrname, rdata, rrtype, sensorID)
212		err := getCounters.MapScan(tmpMap)
213		if err != nil {
214		log.Errorf("getCount: %s", err.Error())
215		continue
216		}
217		out.Count = int(tmpMap["count"].(int64))
218
219		tmpMap = make(map[string]interface{})
220		getFirstSeen := db.Session.Query(`SELECT first_seen FROM observations_firstseen WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?`).Bind(rrname, rdata, rrtype, sensorID)
221		err = getFirstSeen.MapScan(tmpMap)
222		if err != nil {
223		log.Errorf("getFirstSeen: %s", err.Error())
224		continue
225		}
226		out.FirstSeen = int(tmpMap["first_seen"].(int64))
227
228		outs = append(outs, out)
229		} else {
230		log.Warn("result is missing rdata column, something is very wrong")
231		}
232		} else {
233		log.Warn("result is missing rrname column, something is very wrong")
234		}
235
236		}
237
238		return outs, nil
239		}
240
241		// TotalCount returns the overall number of observations across all sensors.
242		func (db *CassandraDB) TotalCount() (int, error) {
243		// TODO
244		return 0, nil
245		}
246
247		// Shutdown closes the database connection, leaving the database unable to
248		// process both reads and writes.
249		func (db *CassandraDB) Shutdown() {
250		close(db.StopChan)
251		if db.Session != nil {
252		db.Session.Close()
253		}
254		}

-134

~~db/db_cassandra_test.go~~ less more

0		package db
1
2		// func TestCassandraSimpleConsume(t *testing.T) {
3		// t.Skip("skipping Cassandra tests due to removed gockle support")
4		// db, m := makeCassandraDBMock()
5		// defer db.Shutdown()
6
7		// inChan := make(chan observation.InputObservation)
8		// stopChan := make(chan bool)
9
10		// go db.ConsumeFeed(inChan)
11
12		// timeStart := time.Now()
13		// timeEnd := time.Now()
14
15		// cnt := 0
16		// var iteratorMock = &gockle.IteratorMock{}
17		// iteratorMock.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
18		// m["rrname"] = "foo.bar"
19		// m["rrtype"] = "A"
20		// m["rdata"] = "12.34.56.78"
21		// m["count"] = 2
22		// m["sensor_id"] = "deadcafe"
23		// cnt++
24		// return cnt < 4
25		// })
26		// iteratorMock.When("Close").Return(nil)
27
28		// cnt2 := 0
29		// var iteratorMockRdata = &gockle.IteratorMock{}
30		// iteratorMockRdata.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
31		// m["rrname"] = "foo.bar"
32		// m["rrtype"] = "A"
33		// m["rdata"] = "12.34.56.79"
34		// m["count"] = 96
35		// m["sensor_id"] = "deadcafe"
36		// cnt2++
37		// return cnt2 < 2
38		// })
39		// iteratorMockRdata.When("Close").Return(nil)
40
41		// cnt3 := 0
42		// var iteratorMockCnt = &gockle.IteratorMock{}
43		// iteratorMockCnt.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
44		// m["count"] = int64(3)
45		// cnt3++
46		// return cnt3 < 2
47		// })
48		// iteratorMockCnt.When("Close").Return(nil)
49
50		// cnt4 := 0
51		// var iteratorMockFirstItem = &gockle.IteratorMock{}
52		// iteratorMockFirstItem.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
53		// m["rrname"] = "foo.bar"
54		// cnt4++
55		// return cnt4 < 2
56		// })
57		// iteratorMockFirstItem.When("Close").Return(nil)
58
59		// m.When("ScanIterator", "SELECT * FROM observations_by_rrname WHERE rrname = ?", mock.Any).Return(iteratorMock)
60		// m.When("Exec", "UPDATE observations_by_rdata SET first_seen = ?, last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
61		// m.When("Exec", "UPDATE observations_by_rdata SET last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
62		// m.When("Exec", "UPDATE observations_by_rrname SET first_seen = ?, last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
63		// m.When("Exec", "UPDATE observations_by_rrname SET last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
64		// m.When("ScanIterator", "SELECT * FROM observations_by_rdata WHERE rdata = ?", []interface{}{"12.34.56.79"}).Return(iteratorMockRdata)
65		// m.When("Exec", "UPDATE observations_counts SET count = count + ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
66		// m.When("ScanIterator", "SELECT rrname FROM observations_by_rrname WHERE rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return(iteratorMockFirstItem)
67		// m.When("ScanIterator", "SELECT count FROM observations_counts WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?", mock.Any).Return(iteratorMockCnt)
68		// m.When("Close").Return()
69
70		// o := observation.InputObservation{
71		// Rrname: "foo.bar",
72		// Rrtype: "A",
73		// SensorID: "deadcafe",
74		// Rdata: "12.34.56.78",
75		// TimestampEnd: timeStart,
76		// TimestampStart: timeEnd,
77		// Count: 1,
78		// }
79		// inChan <- o
80
81		// o = observation.InputObservation{
82		// Rrname: "foo.bar",
83		// Rrtype: "MX",
84		// SensorID: "deadcafe",
85		// Rdata: "12.34.56.77",
86		// TimestampEnd: timeStart,
87		// TimestampStart: timeEnd,
88		// Count: 1,
89		// }
90		// inChan <- o
91
92		// for i := 0; i < 500; i++ {
93		// o = observation.InputObservation{
94		// Rrname: "foo.bar",
95		// Rrtype: "NS",
96		// SensorID: "deadcafe",
97		// Rdata: "12.34.56.79",
98		// TimestampEnd: timeStart,
99		// TimestampStart: timeEnd,
100		// Count: 2,
101		// }
102		// inChan <- o
103		// }
104		// close(stopChan)
105
106		// str := "foo.bar"
107		// obs, err := db.Search(nil, &str, nil, nil)
108		// if err != nil {
109		// t.Fatal(err)
110		// }
111		// if len(obs) != 3 {
112		// t.Fatalf("wrong number of results: %d", len(obs))
113		// }
114
115		// str = "12.34.56.79"
116		// obs, err = db.Search(&str, nil, nil, nil)
117		// if err != nil {
118		// t.Fatal(err)
119		// }
120		// if len(obs) != 1 {
121		// t.Fatalf("wrong number of results: %d", len(obs))
122		// }
123
124		// str = "12.34.56.79"
125		// obs, err = db.Search(&str, nil, nil, &str)
126		// if err != nil {
127		// t.Fatal(err)
128		// }
129		// if len(obs) != 0 {
130		// t.Fatalf("wrong number of results: %d", len(obs))
131		// }
132
133		// }

-94

~~db/db_config.go~~ less more

0		// balboa
1		// Copyright (c) 2018, DCSO GmbH
2
3		package db
4
5		import (
6		"fmt"
7		"runtime"
8
9		log "github.com/sirupsen/logrus"
10		yaml "gopkg.in/yaml.v2"
11		)
12
13		// Setup represents a database backend configuration, typically provided via
14		// YAML.
15		type Setup struct {
16		Database struct {
17		Name string `yaml:"name"`
18		Type string `yaml:"type"`
19		// for local storage
20		DBPath string `yaml:"db_path"`
21		// for RocksDB
22		MemtableMemBudget uint64 `yaml:"membudget"`
23		// for Cassandra
24		Hosts []string `yaml:"hosts"`
25		Username string `yaml:"username"`
26		Password string `yaml:"password"`
27		Nworkers uint `yaml:"nof_workers"`
28		} `yaml:"database"`
29		LoadedDB DB
30		}
31
32		// LoadSetup parses a given YAML description into a new Setup structure.
33		func LoadSetup(in []byte) (*Setup, error) {
34		var s Setup
35		err := yaml.Unmarshal(in, &s)
36		if err != nil {
37		return nil, err
38		}
39		if s.Database.Name == "" {
40		return nil, fmt.Errorf("database name missing")
41		}
42		if s.Database.Type == "" {
43		return nil, fmt.Errorf("database type missing")
44		}
45		switch s.Database.Type {
46		case "rocksdb":
47		if len(s.Database.DBPath) == 0 {
48		return nil, fmt.Errorf("%s: local database path missing", s.Database.Name)
49		}
50		if s.Database.MemtableMemBudget == 0 {
51		log.Infof("%s: memory budget empty, using default of 128MB", s.Database.Name)
52		s.Database.MemtableMemBudget = 128 * 1024 * 1024
53		}
54		case "cassandra":
55		if len(s.Database.Hosts) == 0 {
56		return nil, fmt.Errorf("%s: no Cassandra hosts defined", s.Database.Name)
57		}
58		if s.Database.Nworkers == 0 {
59		s.Database.Nworkers = (uint)(runtime.NumCPU() * 8)
60		log.Infof("%s: number of workers is 0 or undefined, will use default of %d", s.Database.Name, s.Database.Nworkers)
61		}
62		}
63		return &s, nil
64		}
65
66		// Run creates the necessary database objects and makes them ready to
67		// consume or provide data.
68		func (s *Setup) Run() (DB, error) {
69		log.Infof("starting database %s", s.Database.Name)
70		var db DB
71		var err error
72		switch s.Database.Type {
73		case "rocksdb":
74		db, err = MakeRocksDB(s.Database.DBPath, s.Database.MemtableMemBudget)
75		if err != nil {
76		return nil, err
77		}
78		case "cassandra":
79		db, err = MakeCassandraDB(s.Database.Hosts, s.Database.Username, s.Database.Password, s.Database.Nworkers)
80		if err != nil {
81		return nil, err
82		}
83		}
84		s.LoadedDB = db
85		return db, nil
86		}
87
88		// Stop causes a given Setup to shut down the corresponding database
89		// connections defined within.
90		func (s *Setup) Stop(stopChan chan bool) {
91		s.LoadedDB.Shutdown()
92		close(stopChan)
93		}

-105

~~db/db_config_test.go~~ less more

0		package db
1
2		import (
3		"fmt"
4		"io/ioutil"
5		"os"
6		"testing"
7		)
8
9		func TestDBConfigLoad(t *testing.T) {
10		_, err := LoadSetup([]byte("foo["))
11		if err == nil {
12		t.Fatal("did not error with invalid data")
13		}
14
15		_, err = LoadSetup([]byte(""))
16		if err == nil {
17		t.Fatal("did not error with empty config")
18		}
19
20		_, err = LoadSetup([]byte(`
21		database:
22		name: Whatever
23		`))
24		if err == nil {
25		t.Fatal("did not error with missing db type")
26		}
27
28		_, err = LoadSetup([]byte(`
29		database:
30		name: Local RocksDB
31		type: rocksdb
32		`))
33		if err == nil {
34		t.Fatal("did not error with missing path")
35		}
36
37		_, err = LoadSetup([]byte(`
38		database:
39		name: Local RocksDB
40		type: rocksdb
41		db_path: /tmp/balboa
42		`))
43		if err != nil {
44		t.Fatal("does not accept regular config: ", err)
45		}
46
47		_, err = LoadSetup([]byte(`
48		database:
49		name: Local RocksDB
50		type: rocksdb
51		db_path: /tmp/balboa
52		membudget: 1000000
53		`))
54		if err != nil {
55		t.Fatal("does not accept regular config: ", err)
56		}
57
58		_, err = LoadSetup([]byte(`
59		database:
60		name: Local Cassandra
61		type: cassandra
62		hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]
63		`))
64		if err != nil {
65		t.Fatal("does not accept regular config: ", err)
66		}
67
68		_, err = LoadSetup([]byte(`
69		database:
70		name: Local Cassandra
71		type: cassandra
72		`))
73		if err == nil {
74		t.Fatal("did not error with missing hosts")
75		}
76		}
77
78		func TestDBConfigRunRocks(t *testing.T) {
79		dbdir, err := ioutil.TempDir("", "example")
80		if err != nil {
81		t.Fatal(err)
82		}
83		dbs, err := LoadSetup([]byte(fmt.Sprintf(`
84		database:
85		name: Local RocksDB
86		type: rocksdb
87		membudget: 1000000
88		db_path: %s
89		`, dbdir)))
90		if err != nil {
91		t.Fatal("does not accept regular config: ", err)
92		}
93
94		_, err = dbs.Run()
95		if err != nil {
96		t.Fatal(err)
97		}
98		defer os.RemoveAll(dbdir)
99
100		stopChan := make(chan bool)
101		dbs.Stop(stopChan)
102		<-stopChan
103
104		}

-189

~~db/db_mysql.go~~ less more

0		// balboa
1		// Copyright (c) 2018, DCSO GmbH
2
3		package db
4
5		import (
6		"database/sql"
7		"time"
8
9		"github.com/DCSO/balboa/observation"
10
11		// imported for side effects
12		_ "github.com/go-sql-driver/mysql"
13		log "github.com/sirupsen/logrus"
14		)
15
16		// create table observations (
17		// id bigint AUTO_INCREMENT,
18		// sensorID varchar(255) NOT NULL,
19		// rrname varchar(255) NOT NULL,
20		// rdata varchar(255) NOT NULL,
21		// rrcode varchar(64) NULL,
22		// rrtype varchar(64) NULL,
23		// count int NOT NULL,
24		// time_first timestamp NOT NULL,
25		// time_last timestamp NOT NULL,
26		// PRIMARY KEY (id),
27		// UNIQUE KEY (rdata, rrname, sensorID),
28		// INDEX rrname (rrname)
29		// );
30
31		// MySQLDB is a DB implementation based on MySQL.
32		type MySQLDB struct {
33		DB *sql.DB
34		}
35
36		const (
37		mysqlbufsize = 50
38		upsertSQL = `
39		INSERT INTO observations
40		(sensorID, rrname, rdata, rrcode, rrtype, count, time_first, time_last)
41		VALUES
42		(?, ?, ?, ?, ?, ?, ?, ?)
43		ON DUPLICATE KEY UPDATE
44		count = count + VALUES(count),
45		time_last = VALUES(time_last);`
46		qrySQL = `
47		SELECT
48		sensorID, rrname, rdata, rrtype, count, time_first, time_last
49		FROM
50		observations
51		WHERE
52		rdata like ? AND rrname like ? AND rrtype like ? AND sensorID like ?;`
53		)
54
55		// MakeMySQLDB returns a new MySQLDB instance based on the given sql.DB
56		// pointer.
57		func MakeMySQLDB(pdb sql.DB) MySQLDB {
58		db := &MySQLDB{
59		DB: pdb,
60		}
61		return db
62		}
63
64		// AddObservation adds a single observation synchronously to the database.
65		func (db *MySQLDB) AddObservation(obs observation.InputObservation) observation.Observation {
66		//Not implemented
67		log.Warn("AddObservation() not implemented on MySQL DB backend")
68		return observation.Observation{}
69		}
70
71		// ConsumeFeed accepts observations from a channel and queues them for
72		// database insertion.
73		func (db *MySQLDB) ConsumeFeed(inChan chan observation.InputObservation, stopChan chan bool) {
74		buf := make([]observation.InputObservation, mysqlbufsize)
75		i := 0
76		for {
77		select {
78		case <-stopChan:
79		log.Info("database ingest terminated")
80		return
81		default:
82		if i < mysqlbufsize {
83		log.Debug("buffering ", i)
84		buf[i] = <-inChan
85		i++
86		} else {
87		// flushing buffer in one transaction
88		i = 0
89		startTime := time.Now()
90		tx, err := db.DB.Begin()
91		if err != nil {
92		log.Warn(err)
93		}
94		var upsert *sql.Stmt
95		upsert, _ = tx.Prepare(upsertSQL)
96		for _, obs := range buf {
97		_, err := upsert.Exec(obs.SensorID, obs.Rrname,
98		obs.Rdata, nil, obs.Rrtype, obs.Count,
99		obs.TimestampStart, obs.TimestampEnd)
100		if err != nil {
101		log.Warn(err)
102		tx.Rollback()
103		}
104		}
105		tx.Commit()
106		log.Infof("insert Tx took %v", time.Since(startTime))
107		}
108		}
109		}
110		}
111
112		// Search returns a slice of observations matching one or more criteria such
113		// as rdata, rrname, rrtype or sensor ID.
114		func (db MySQLDB) Search(qrdata, qrrname, qrrtype, qsensorID string) ([]observation.Observation, error) {
115		outs := make([]observation.Observation, 0)
116		stmt, err := db.DB.Prepare(qrySQL)
117		if err != nil {
118		return nil, err
119		}
120
121		rdata := "%"
122		if qrdata != nil {
123		rdata = *qrdata
124		}
125		sensorID := "%"
126		if qsensorID != nil {
127		sensorID = *qsensorID
128		}
129		rrname := "%"
130		if qrrname != nil {
131		rrname = *qrrname
132		}
133		rrtype := "%"
134		if qrrtype != nil {
135		rrtype = *qrrtype
136		}
137
138		rows, err := stmt.Query(rdata, rrname, rrtype, sensorID)
139		if err != nil {
140		return nil, err
141		}
142
143		for rows.Next() {
144		out := observation.Observation{}
145		var lastseen, firstseen time.Time
146		err := rows.Scan(&out.SensorID, &out.RRName, &out.RData, &out.RRType,
147		&out.Count, &firstseen, &lastseen)
148		if err != nil {
149		return nil, err
150		}
151		out.FirstSeen = int(firstseen.Unix())
152		out.LastSeen = int(lastseen.Unix())
153		outs = append(outs, out)
154		}
155		err = rows.Err()
156		if err != nil {
157		return nil, err
158		}
159		return outs, nil
160		}
161
162		// TotalCount returns the overall number of observations across all sensors.
163		func (db *MySQLDB) TotalCount() (int, error) {
164		var val int
165		var err error
166		rows, err := db.DB.Query("SELECT count(*) FROM observations")
167		if err != nil {
168		return 0, err
169		}
170		defer rows.Close()
171		for rows.Next() {
172		err := rows.Scan(&val)
173		if err != nil {
174		return 0, err
175		}
176		}
177		err = rows.Err()
178		if err != nil {
179		return 0, err
180		}
181		return val, nil
182		}
183
184		// Shutdown closes the database connection, leaving the database unable to
185		// process both reads and writes.
186		func (db *MySQLDB) Shutdown() {
187		db.DB.Close()
188		}

-152

~~db/db_mysql_test.go~~ less more

0		package db
1
2		import (
3		"testing"
4		"time"
5
6		sqlmock "github.com/DATA-DOG/go-sqlmock"
7		"github.com/DCSO/balboa/observation"
8		)
9
10		func createMysqlTmpDB(t testing.T) (MySQLDB, sqlmock.Sqlmock) {
11		db, mock, err := sqlmock.New()
12		if err != nil {
13		t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
14		}
15
16		return MakeMySQLDB(db), mock
17		}
18
19		func TestMySQLSimpleConsume(t *testing.T) {
20		db, mock := createMysqlTmpDB(t)
21		defer db.Shutdown()
22
23		inChan := make(chan observation.InputObservation)
24		stopChan := make(chan bool)
25
26		go db.ConsumeFeed(inChan, stopChan)
27
28		timeStart := time.Now()
29		timeEnd := time.Now()
30		mock.ExpectBegin()
31		mock.ExpectPrepare("INSERT INTO observations")
32		mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "x.foo.bar", "12.34.56.78", nil, "A", 1, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
33		mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "y.foo.bar", "12.34.56.77", nil, "MX", 1, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
34		for i := 0; i < 48; i++ {
35		mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "foo.bar", "12.34.56.79", nil, "NS", 2, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
36		}
37		mock.ExpectPrepare("SELECT (.+) FROM observations")
38		retRows := sqlmock.NewRows(
39		[]string{
40		"sensor_id",
41		"rrname",
42		"rdata",
43		"rrtype",
44		"count",
45		"time_first",
46		"time_last",
47		},
48		)
49		retRows.AddRow("12.34.56.78", "A", "x.foo.bar", "deadcafe", 1, timeStart, timeEnd)
50		retRows.AddRow("12.34.56.78", "NX", "y.foo.bar", "deadcafe", 1, timeStart, timeEnd)
51		retRows.AddRow("12.34.56.78", "NS", "foo.bar", "deadcafe", 96, timeStart, timeEnd)
52		mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
53		mock.ExpectPrepare("SELECT (.+) FROM observations")
54		retRows = sqlmock.NewRows(
55		[]string{
56		"sensor_id",
57		"rrname",
58		"rdata",
59		"rrtype",
60		"count",
61		"time_first",
62		"time_last",
63		},
64		)
65		retRows.AddRow("12.34.56.78", "NS", "foo.bar", "deadcafe", 96, timeStart, timeEnd)
66		mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
67		mock.ExpectPrepare("SELECT (.+) FROM observations")
68		retRows = sqlmock.NewRows(
69		[]string{
70		"sensor_id",
71		"rrname",
72		"rdata",
73		"rrtype",
74		"count",
75		"time_first",
76		"time_last",
77		},
78		)
79		mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
80		mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(sqlmock.NewRows(
81		[]string{
82		"count",
83		}).AddRow(3))
84
85		inChan <- observation.InputObservation{
86		Rrname: "x.foo.bar",
87		Rrtype: "A",
88		SensorID: "deadcafe",
89		Rdata: "12.34.56.78",
90		TimestampEnd: timeEnd,
91		TimestampStart: timeStart,
92		Count: 1,
93		}
94		inChan <- observation.InputObservation{
95		Rrname: "y.foo.bar",
96		Rrtype: "MX",
97		SensorID: "deadcafe",
98		Rdata: "12.34.56.77",
99		TimestampEnd: timeEnd,
100		TimestampStart: timeStart,
101		Count: 1,
102		}
103		for i := 0; i < 50; i++ {
104		inChan <- observation.InputObservation{
105		Rrname: "foo.bar",
106		Rrtype: "NS",
107		SensorID: "deadcafe",
108		Rdata: "12.34.56.79",
109		TimestampEnd: timeEnd,
110		TimestampStart: timeStart,
111		Count: 2,
112		}
113		}
114		close(stopChan)
115
116		str := "foo.bar"
117		obs, err := db.Search(nil, &str, nil, nil)
118		if err != nil {
119		t.Fatal(err)
120		}
121		if len(obs) != 3 {
122		t.Fatalf("wrong number of results: %d", len(obs))
123		}
124
125		str = "12.34.56.79"
126		obs, err = db.Search(&str, nil, nil, nil)
127		if err != nil {
128		t.Fatal(err)
129		}
130		if len(obs) != 1 {
131		t.Fatalf("wrong number of results: %d", len(obs))
132		}
133
134		str = "12.34.56.79"
135		obs, err = db.Search(&str, nil, nil, &str)
136		if err != nil {
137		t.Fatal(err)
138		}
139		if len(obs) != 0 {
140		t.Fatalf("wrong number of results: %d", len(obs))
141		}
142
143		tc, err := db.TotalCount()
144		if err != nil {
145		t.Fatal(err)
146		}
147		if tc != 3 {
148		t.Fatalf("wrong number of results: %d", len(obs))
149		}
150
151		}

+107

-0

db/db_pool.go less more

	0	// balboa-backends
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	// based on: https://github.com/go-baa/pool/blob/master/pool.go
	4
	5	package db
	6
	7	import (
	8	"fmt"
	9	"net"
	10	"sync"
	11	)
	12
	13	type Pool struct {
	14	low int
	15	Connect func() (net.Conn, error)
	16	Ping func(interface{}) bool
	17	Close func(net.Conn)
	18	store chan net.Conn
	19	destroyMutex sync.Mutex
	20	}
	21
	22	func MakePool(initCap, maxCap int, connectFn func() (net.Conn, error)) (*Pool, error) {
	23	if maxCap == 0 \|\| initCap > maxCap {
	24	return nil, fmt.Errorf("invalid capacity settings")
	25	}
	26	p := new(Pool)
	27	p.store = make(chan net.Conn, maxCap)
	28	p.low = initCap
	29	if connectFn != nil {
	30	p.Connect = connectFn
	31	}
	32	for i := 0; i < initCap; i++ {
	33	v, err := p.create()
	34	if err != nil {
	35	return p, err
	36	}
	37	p.store <- v
	38	}
	39	return p, nil
	40	}
	41
	42	func (p *Pool) Len() int {
	43	return len(p.store)
	44	}
	45
	46	func (p *Pool) Refill() {
	47	for i := len(p.store); i < p.low; i++ {
	48	v, err := p.create()
	49	if err != nil {
	50	return
	51	}
	52	p.store <- v
	53	}
	54	}
	55
	56	func (p *Pool) Get() (net.Conn, error) {
	57	if p.store == nil {
	58	return p.create()
	59	}
	60	for {
	61	select {
	62	case v := <-p.store:
	63	if p.Ping != nil && !p.Ping(v) {
	64	continue
	65	}
	66	return v, nil
	67	default:
	68	return p.create()
	69	}
	70	}
	71	}
	72
	73	func (p *Pool) Put(v net.Conn) {
	74	select {
	75	case p.store <- v:
	76	return
	77	default:
	78	if p.Close != nil {
	79	p.Close(v)
	80	}
	81	return
	82	}
	83	}
	84
	85	func (p *Pool) Teardown() {
	86	p.destroyMutex.Lock()
	87	defer p.destroyMutex.Unlock()
	88	if p.store == nil {
	89	// pool already destroyed
	90	return
	91	}
	92	close(p.store)
	93	for v := range p.store {
	94	if p.Close != nil {
	95	p.Close(v)
	96	}
	97	}
	98	p.store = nil
	99	}
	100
	101	func (p *Pool) create() (net.Conn, error) {
	102	if p.Connect == nil {
	103	return nil, fmt.Errorf("Pool.Connect is nil, can not create connection")
	104	}
	105	return p.Connect()
	106	}

+295

-0

db/db_protocol.go less more

	0	// balboa
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	package db
	4
	5	import (
	6	"bytes"
	7	"errors"
	8	"net"
	9
	10	obs "github.com/DCSO/balboa/observation"
	11
	12	log "github.com/sirupsen/logrus"
	13	"github.com/ugorji/go/codec"
	14	)
	15
	16	const (
	17	TypeInputRequest = 1
	18	TypeQueryRequest = 2
	19	TypeBackupRequest = 3
	20	TypeDumpRequest = 4
	21	TypeErrorResponse = 128
	22	TypeQueryResponse = 129
	23	TypeQueryStreamStartResponse = 130
	24	TypeQueryStreamDataResponse = 131
	25	TypeQueryStreamEndResponse = 132
	26	)
	27
	28	type TypedMessage struct {
	29	Type uint8 `codec:"T"`
	30	EncodedMessage []byte `codec:"M"`
	31	}
	32
	33	type BackupRequest struct {
	34	Path string `codec:"P"`
	35	}
	36
	37	type DumpRequest struct {
	38	Path string `codec:"P"`
	39	}
	40
	41	type QueryRequest struct {
	42	Qrdata, Qrrname, Qrrtype, QsensorID string
	43	Hrdata, Hrrname, Hrrtype, HsensorID bool
	44	Limit int
	45	}
	46
	47	type QueryResponse struct {
	48	Obs []obs.Observation
	49	}
	50
	51	type ErrorResponse struct {
	52	Message string
	53	}
	54
	55	type Encoder struct {
	56	inner *bytes.Buffer
	57	outer *bytes.Buffer
	58	enc *codec.Encoder
	59	}
	60
	61	type Decoder struct {
	62	conn net.Conn
	63	outer_dec *codec.Decoder
	64	inner_dec *codec.Decoder
	65	}
	66
	67	func (enc *Encoder) Release() {
	68	enc.enc.Release()
	69	}
	70
	71	func MakeEncoder() *Encoder {
	72	inner := new(bytes.Buffer)
	73	outer := new(bytes.Buffer)
	74	h := new(codec.MsgpackHandle)
	75	h.ExplicitRelease = true
	76	h.WriteExt = true
	77	enc := codec.NewEncoder(inner, h)
	78	return &Encoder{inner: inner, outer: outer, enc: enc}
	79	}
	80
	81	func MakeDecoder(conn net.Conn) *Decoder {
	82	outer_h := new(codec.MsgpackHandle)
	83	outer_h.ExplicitRelease = true
	84	outer_h.WriteExt = true
	85	outer_dec := codec.NewDecoder(conn, outer_h)
	86	inner_h := new(codec.MsgpackHandle)
	87	inner_h.ExplicitRelease = true
	88	inner_h.WriteExt = true
	89	inner_dec := codec.NewDecoder(new(bytes.Buffer), inner_h)
	90	return &Decoder{inner_dec: inner_dec, outer_dec: outer_dec, conn: conn}
	91	}
	92
	93	func (dec *Decoder) Release() {
	94	dec.inner_dec.Release()
	95	dec.outer_dec.Release()
	96	}
	97
	98	func (dec Decoder) ExpectQueryResponse() (QueryResponse, error) {
	99	msg, msg_err := dec.ExpectTypedMessage()
	100	if msg_err != nil {
	101	return nil, msg_err
	102	}
	103	dec.inner_dec.Reset(bytes.NewBuffer(msg.EncodedMessage))
	104	if msg.Type == TypeErrorResponse {
	105	var rep ErrorResponse
	106	inner_err := dec.inner_dec.Decode(&rep)
	107	if inner_err != nil {
	108	return nil, inner_err
	109	}
	110	return nil, errors.New(rep.Message)
	111	}
	112	if msg.Type == TypeQueryResponse {
	113	var rep QueryResponse
	114	inner_err := dec.inner_dec.Decode(&rep)
	115	if inner_err != nil {
	116	return nil, inner_err
	117	}
	118	return &rep, nil
	119	} else if msg.Type == TypeQueryStreamStartResponse {
	120	log.Debugf("got stream start response")
	121	return dec.ExpectQueryStreamResponse()
	122	} else {
	123	return nil, errors.New("invalid query response")
	124	}
	125	}
	126
	127	func (enc Encoder) EncodeInputRequest(o obs.InputObservation) (bytes.Buffer, error) {
	128	enc.inner.Reset()
	129	enc.outer.Reset()
	130	enc.enc.Reset(enc.inner)
	131	inner_err := enc.enc.Encode(o)
	132	if inner_err != nil {
	133	return nil, inner_err
	134	}
	135	enc.enc.Reset(enc.outer)
	136	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeInputRequest, EncodedMessage: enc.inner.Bytes()})
	137	if outer_err != nil {
	138	return nil, outer_err
	139	}
	140	return enc.outer, nil
	141	}
	142
	143	func (enc Encoder) EncodeQueryRequest(qry QueryRequest) (bytes.Buffer, error) {
	144	enc.inner.Reset()
	145	enc.outer.Reset()
	146	enc.enc.Reset(enc.inner)
	147	inner_err := enc.enc.Encode(qry)
	148	if inner_err != nil {
	149	return nil, inner_err
	150	}
	151	enc.enc.Reset(enc.outer)
	152	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeQueryRequest, EncodedMessage: enc.inner.Bytes()})
	153	if outer_err != nil {
	154	return nil, outer_err
	155	}
	156	return enc.outer, nil
	157	}
	158
	159	func (enc Encoder) EncodeQueryStreamStartResponse() (bytes.Buffer, error) {
	160	enc.inner.Reset()
	161	enc.outer.Reset()
	162	enc.enc.Reset(enc.inner)
	163	enc.enc.Reset(enc.outer)
	164	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeQueryStreamStartResponse, EncodedMessage: enc.inner.Bytes()})
	165	if outer_err != nil {
	166	return nil, outer_err
	167	}
	168	return enc.outer, nil
	169	}
	170
	171	func (enc Encoder) EncodeQueryStreamEndResponse() (bytes.Buffer, error) {
	172	enc.inner.Reset()
	173	enc.outer.Reset()
	174	enc.enc.Reset(enc.inner)
	175	enc.enc.Reset(enc.outer)
	176	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeQueryStreamEndResponse, EncodedMessage: enc.inner.Bytes()})
	177	if outer_err != nil {
	178	return nil, outer_err
	179	}
	180	return enc.outer, nil
	181	}
	182
	183	func (enc Encoder) EncodeQueryStreamDataResponse(entry obs.Observation) (bytes.Buffer, error) {
	184	enc.inner.Reset()
	185	enc.outer.Reset()
	186	enc.enc.Reset(enc.inner)
	187	inner_err := enc.enc.Encode(entry)
	188	if inner_err != nil {
	189	return nil, inner_err
	190	}
	191	enc.enc.Reset(enc.outer)
	192	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeQueryStreamDataResponse, EncodedMessage: enc.inner.Bytes()})
	193	if outer_err != nil {
	194	return nil, outer_err
	195	}
	196	return enc.outer, nil
	197	}
	198
	199	func (dec Decoder) ExpectInputRequestFromBytes(buf []byte) (obs.InputObservation, error) {
	200	dec.inner_dec.Reset(bytes.NewBuffer(buf))
	201	var msg obs.InputObservation
	202	err := dec.inner_dec.Decode(&msg)
	203	if err != nil {
	204	return nil, err
	205	}
	206	return &msg, nil
	207	}
	208
	209	func (enc Encoder) EncodeErrorResponse(err ErrorResponse) (bytes.Buffer, error) {
	210	enc.inner.Reset()
	211	enc.outer.Reset()
	212	inner_err := enc.enc.Encode(err)
	213	if inner_err != nil {
	214	return nil, inner_err
	215	}
	216	enc.enc.Reset(enc.outer)
	217	outer_err := enc.enc.Encode(&TypedMessage{Type: TypeQueryRequest, EncodedMessage: enc.inner.Bytes()})
	218	if outer_err != nil {
	219	return nil, outer_err
	220	}
	221	return enc.outer, nil
	222	}
	223
	224	func (dec Decoder) ExpectTypedMessage() (TypedMessage, error) {
	225	var msg TypedMessage
	226	err := dec.outer_dec.Decode(&msg)
	227	if err != nil {
	228	return nil, err
	229	}
	230	return &msg, nil
	231	}
	232
	233	func (dec Decoder) ExpectQueryRequestFromBytes(buf []byte) (QueryRequest, error) {
	234	dec.inner_dec.Reset(bytes.NewBuffer(buf))
	235	var msg QueryRequest
	236	err := dec.inner_dec.Decode(&msg)
	237	if err != nil {
	238	return nil, err
	239	}
	240	return &msg, nil
	241	}
	242
	243	func (dec Decoder) ExpectBackupRequestFromBytes(buf []byte) (BackupRequest, error) {
	244	dec.inner_dec.Reset(bytes.NewBuffer(buf))
	245	var msg BackupRequest
	246	err := dec.inner_dec.Decode(&msg)
	247	if err != nil {
	248	return nil, err
	249	}
	250	return &msg, nil
	251	}
	252
	253	func (dec Decoder) ExpectDumpRequestFromBytes(buf []byte) (DumpRequest, error) {
	254	dec.inner_dec.Reset(bytes.NewBuffer(buf))
	255	var msg DumpRequest
	256	err := dec.inner_dec.Decode(&msg)
	257	if err != nil {
	258	return nil, err
	259	}
	260	return &msg, nil
	261	}
	262
	263	func (dec Decoder) ExpectQueryStreamResponse() (QueryResponse, error) {
	264	var res []obs.Observation
	265	for {
	266	msg, msg_err := dec.ExpectTypedMessage()
	267	if msg_err != nil {
	268	return nil, msg_err
	269	}
	270	dec.inner_dec.Reset(bytes.NewBuffer(msg.EncodedMessage))
	271	if msg.Type == TypeErrorResponse {
	272	var rep ErrorResponse
	273	inner_err := dec.inner_dec.Decode(&rep)
	274	if inner_err != nil {
	275	log.Warnf("got error response during stream data")
	276	return nil, inner_err
	277	}
	278	return nil, errors.New(rep.Message)
	279	} else if msg.Type == TypeQueryStreamDataResponse {
	280	var rep obs.Observation
	281	inner_err := dec.inner_dec.Decode(&rep)
	282	if inner_err != nil {
	283	log.Warnf("decoding stream data response failed")
	284	return nil, inner_err
	285	}
	286	res = append(res, rep)
	287	} else if msg.Type == TypeQueryStreamEndResponse {
	288	return &QueryResponse{Obs: res}, nil
	289	} else {
	290	log.Warnf("invalid message type `%v`", msg.Type)
	291	return nil, errors.New("received invalid message type from backend")
	292	}
	293	}
	294	}

+148

-0

db/db_remote.go less more

	0	// balboa
	1	// Copyright (c) 2019, DCSO GmbH
	2
	3	package db
	4
	5	import (
	6	"errors"
	7	"net"
	8
	9	obs "github.com/DCSO/balboa/observation"
	10
	11	log "github.com/sirupsen/logrus"
	12	)
	13
	14	type RemoteBackend struct {
	15	host string
	16	StopChan chan bool
	17	}
	18
	19	func MakeRemoteBackend(host string, refill bool) (*RemoteBackend, error) {
	20	return &RemoteBackend{StopChan: make(chan bool), host: host}, nil
	21	}
	22
	23	func (db *RemoteBackend) AddObservation(o obs.InputObservation) obs.Observation {
	24	log.Warn("AddObservation() not implemented")
	25	return obs.Observation{}
	26	}
	27
	28	func (db *RemoteBackend) Backup(path string) {
	29	log.Warnf("backup request unimplemented")
	30	}
	31
	32	func (db *RemoteBackend) Dump(path string) {
	33	log.Warnf("dump request unimplemented")
	34	}
	35
	36	func (db *RemoteBackend) ConsumeFeed(inChan chan obs.InputObservation) {
	37	enc := MakeEncoder()
	38	defer enc.Release()
	39	conn, conn_err := net.Dial("tcp", db.host)
	40	if conn_err != nil {
	41	log.Warnf("connecting to backend failed: %v", conn_err)
	42	return
	43	}
	44	defer conn.Close()
	45	for {
	46	select {
	47	case <-db.StopChan:
	48	log.Info("stop request received")
	49	return
	50	case obs := <-inChan:
	51	log.Debug("received observation")
	52	w, err := enc.EncodeInputRequest(obs)
	53	if err != nil {
	54	log.Warnf("encoding observation failed: %s", err)
	55	continue
	56	}
	57	wanted := w.Len()
	58	n, err := w.WriteTo(conn)
	59	if err != nil {
	60	log.Warnf("sending observation failed: %s", err)
	61	return
	62	}
	63	if n != int64(wanted) {
	64	log.Warnf("short write")
	65	return
	66	}
	67	}
	68	}
	69	}
	70
	71	func sanitize(s *string) string {
	72	if s == nil {
	73	return ""
	74	} else {
	75	return *s
	76	}
	77	}
	78
	79	func (db RemoteBackend) Search(qrdata, qrrname, qrrtype, qsensorID string, limit int) ([]obs.Observation, error) {
	80	qry := QueryRequest{
	81	Qrdata: sanitize(qrdata),
	82	Hrdata: qrdata != nil,
	83	Qrrname: sanitize(qrrname),
	84	Hrrname: qrrname != nil,
	85	Qrrtype: sanitize(qrrtype),
	86	Hrrtype: qrrtype != nil,
	87	QsensorID: sanitize(qsensorID),
	88	HsensorID: qsensorID != nil,
	89	Limit: limit,
	90	}
	91
	92	conn, conn_err := net.Dial("tcp", db.host)
	93	if conn_err != nil {
	94	log.Warnf("connecting to backend failed %v", conn_err)
	95	return []obs.Observation{}, conn_err
	96	}
	97	defer conn.Close()
	98
	99	enc := MakeEncoder()
	100	defer enc.Release()
	101
	102	w, enc_err := enc.EncodeQueryRequest(qry)
	103	if enc_err != nil {
	104	log.Warnf("unable to encode query")
	105	return []obs.Observation{}, enc_err
	106	}
	107
	108	wanted := w.Len()
	109
	110	n, err_wr := w.WriteTo(conn)
	111	if err_wr != nil {
	112	log.Infof("sending query failed; closing connection")
	113	conn.Close()
	114	return []obs.Observation{}, err_wr
	115	}
	116
	117	if n != int64(wanted) {
	118	log.Infof("sending query failed; short write; closing connection")
	119	conn.Close()
	120	return []obs.Observation{}, errors.New("sending query failed")
	121	}
	122
	123	log.Debugf("sent query (%d bytes)", n)
	124
	125	dec := MakeDecoder(conn)
	126	defer dec.Release()
	127
	128	result, err := dec.ExpectQueryResponse()
	129
	130	log.Debugf("received answer")
	131
	132	if err != nil {
	133	log.Warnf("ExpectQueryResponse() failed with `%v`", err)
	134	conn.Close()
	135	return []obs.Observation{}, err
	136	}
	137
	138	return result.Obs, nil
	139	}
	140
	141	func (db *RemoteBackend) TotalCount() (int, error) {
	142	return -1, nil
	143	}
	144
	145	func (db *RemoteBackend) Shutdown() {
	146	close(db.StopChan)
	147	}

-312

~~db/db_rocksdb.go~~ less more

0		// balboa
1		// Copyright (c) 2018, DCSO GmbH
2
3		package db
4
5		// #cgo LDFLAGS: -lrocksdb -ltpl
6		// #cgo CFLAGS: -O3 -Wno-implicit-function-declaration -Wall -Wextra -Wno-unused-parameter
7		// #include "obs_rocksdb.h"
8		// #include <stdlib.h>
9		import "C"
10
11		import (
12		"encoding/json"
13		"fmt"
14		"strings"
15		"time"
16		"unsafe"
17
18		"github.com/DCSO/balboa/observation"
19
20		log "github.com/sirupsen/logrus"
21		)
22
23		const (
24		rocksTxSize = 50
25		keySepChar = "\x1f" // we use ASCII Unit separators
26		)
27
28		// RocksDB is a DB implementation based on Facebook's RocksDB engine.
29		type RocksDB struct {
30		db *C.ObsDB
31		stopChan chan bool
32		}
33
34		func rocksMakeKey(sensor string, rrname string, rrtype string, rdata string) string {
35		k := fmt.Sprintf("o%s%s%s%s%s%s%s%s", keySepChar, rrname, keySepChar, sensor, keySepChar, rrtype, keySepChar, rdata)
36		return k
37		}
38
39		func rocksMakeInvKey(sensor string, rrname string, rrtype string, rdata string) string {
40		k := fmt.Sprintf("i%s%s%s%s%s%s%s%s", keySepChar, rdata, keySepChar, sensor, keySepChar, rrname, keySepChar, rrtype)
41		return k
42		}
43
44		// MakeRocksDB returns a new RocksDB instance, storing data at the given dbPath.
45		// The second parameter specifies a memory budget which determines the size of
46		// memtables and caches.
47		func MakeRocksDB(dbPath string, membudget uint64) (*RocksDB, error) {
48		log.Info("opening database...")
49
50		e := C.error_new()
51		defer C.error_delete(e)
52		cdbPath := C.CString(dbPath)
53		cdb := C.obsdb_open(cdbPath, C.size_t(membudget), e)
54		defer C.free(unsafe.Pointer(cdbPath))
55
56		if C.error_is_set(e) {
57		return nil, fmt.Errorf("%s", C.GoString(C.error_get(e)))
58		}
59
60		db := &RocksDB{
61		db: cdb,
62		stopChan: make(chan bool),
63		}
64
65		log.Info("database ready")
66		return db, nil
67		}
68
69		// MakeRocksDBReadonly returns a new read-only RocksDB instance.
70		func MakeRocksDBReadonly(dbPath string) (*RocksDB, error) {
71		log.Info("opening database...")
72
73		e := C.error_new()
74		defer C.error_delete(e)
75		cdbPath := C.CString(dbPath)
76		cdb := C.obsdb_open_readonly(cdbPath, e)
77		defer C.free(unsafe.Pointer(cdbPath))
78
79		if C.error_is_set(e) {
80		return nil, fmt.Errorf("%s", C.GoString(C.error_get(e)))
81		}
82
83		db := &RocksDB{
84		db: cdb,
85		stopChan: make(chan bool),
86		}
87
88		log.Info("database ready")
89		return db, nil
90		}
91
92		func rocksTxDedup(in []observation.InputObservation) []observation.InputObservation {
93		cache := make(map[string]*observation.InputObservation)
94		for i, inObs := range in {
95		key := rocksMakeKey(inObs.SensorID, inObs.Rrname, inObs.Rrtype, inObs.Rdata)
96		_, ok := cache[key]
97		if ok {
98		cache[key].Count += inObs.Count
99		cache[key].TimestampEnd = inObs.TimestampEnd
100		} else {
101		cache[key] = &in[i]
102		}
103		}
104		out := make([]observation.InputObservation, 0)
105		for _, v := range cache {
106		out = append(out, *v)
107		}
108		if len(in) != len(out) {
109		log.Debugf("TX dedup: %d -> %d", len(in), len(out))
110		}
111		return out
112		}
113
114		// AddObservation adds a single observation synchronously to the database.
115		func (db *RocksDB) AddObservation(obs observation.InputObservation) observation.Observation {
116		var cobs C.Observation
117		e := C.error_new()
118		defer C.error_delete(e)
119
120		key := rocksMakeKey(obs.SensorID, obs.Rrname, obs.Rrtype,
121		obs.Rdata)
122		invKey := rocksMakeInvKey(obs.SensorID, obs.Rrname,
123		obs.Rrtype, obs.Rdata)
124
125		cobs.key = C.CString(key)
126		cobs.inv_key = C.CString(invKey)
127		cobs.count = C.uint(obs.Count)
128		cobs.last_seen = C.uint(obs.TimestampEnd.Unix())
129		cobs.first_seen = C.uint(obs.TimestampStart.Unix())
130
131		C.obsdb_put(db.db, &cobs, e)
132		if C.error_is_set(e) {
133		log.Errorf("%s", C.GoString(C.error_get(e)))
134		}
135
136		C.free(unsafe.Pointer(cobs.key))
137		C.free(unsafe.Pointer(cobs.inv_key))
138
139		r, err := db.Search(&obs.Rdata, &obs.Rrname, &obs.Rrtype, &obs.SensorID)
140		if err != nil {
141		log.Error(err)
142		}
143		return r[0]
144		}
145
146		type rocksObservation struct {
147		Count int
148		LastSeen int64
149		FirstSeen int64
150		}
151
152		// ConsumeFeed accepts observations from a channel and queues them for
153		// database insertion.
154		func (db *RocksDB) ConsumeFeed(inChan chan observation.InputObservation) {
155		var i = 0
156		buf := make([]observation.InputObservation, rocksTxSize)
157		var cobs C.Observation
158		e := C.error_new()
159		defer C.error_delete(e)
160
161		for {
162		select {
163		case <-db.stopChan:
164		log.Info("database ingest terminated")
165		return
166		default:
167		obs := <-inChan
168		if i < rocksTxSize {
169		buf[i] = obs
170		i++
171		} else {
172		i = 0
173		startTime := time.Now()
174		for _, obs := range rocksTxDedup(buf) {
175		if obs.Rrtype == "" {
176		continue
177		}
178
179		key := rocksMakeKey(obs.SensorID, obs.Rrname, obs.Rrtype,
180		obs.Rdata)
181		invKey := rocksMakeInvKey(obs.SensorID, obs.Rrname,
182		obs.Rrtype, obs.Rdata)
183
184		cobs.key = C.CString(key)
185		cobs.inv_key = C.CString(invKey)
186		cobs.count = C.uint(obs.Count)
187		cobs.last_seen = C.uint(obs.TimestampEnd.Unix())
188		cobs.first_seen = C.uint(obs.TimestampStart.Unix())
189
190		C.obsdb_put(db.db, &cobs, e)
191		if C.error_is_set(e) {
192		log.Errorf("%s", C.GoString(C.error_get(e)))
193		}
194
195		C.free(unsafe.Pointer(cobs.key))
196		C.free(unsafe.Pointer(cobs.inv_key))
197		}
198		log.Debugf("insert Tx took %v", time.Since(startTime))
199		}
200		}
201		}
202		}
203
204		//export cgoLogInfo
205		func cgoLogInfo(str *C.char) {
206		log.Info(C.GoString(str))
207		}
208
209		//export cgoLogDebug
210		func cgoLogDebug(str *C.char) {
211		log.Debug(C.GoString(str))
212		}
213
214		// Search returns a slice of observations matching one or more criteria such
215		// as rdata, rrname, rrtype or sensor ID.
216		func (db RocksDB) Search(qrdata, qrrname, qrrtype, qsensorID string) ([]observation.Observation, error) {
217		outs := make([]observation.Observation, 0)
218		var cqrdata, cqrrname, cqrrtype, cqsensorID *C.char
219		var i uint
220
221		if qrdata == nil {
222		cqrdata = nil
223		} else {
224		cqrdata = C.CString(*qrdata)
225		defer C.free(unsafe.Pointer(cqrdata))
226		}
227		if qrrname == nil {
228		cqrrname = nil
229		} else {
230		cqrrname = C.CString(*qrrname)
231		defer C.free(unsafe.Pointer(cqrrname))
232		}
233		if qrrtype == nil {
234		cqrrtype = nil
235		} else {
236		cqrrtype = C.CString(*qrrtype)
237		defer C.free(unsafe.Pointer(cqrrtype))
238		}
239		if qsensorID == nil {
240		cqsensorID = nil
241		} else {
242		cqsensorID = C.CString(*qsensorID)
243		defer C.free(unsafe.Pointer(cqsensorID))
244		}
245
246		r := C.obsdb_search(db.db, cqrdata, cqrrname, cqrrtype, cqsensorID)
247		defer C.obs_set_delete(r)
248
249		for i = 0; i < uint(C.obs_set_size(r)); i++ {
250		o := C.obs_set_get(r, C.ulong(i))
251		valArr := strings.Split(C.GoString(o.key), keySepChar)
252		outs = append(outs, observation.Observation{
253		SensorID: valArr[2],
254		Count: int(o.count),
255		RData: valArr[4],
256		RRName: valArr[1],
257		RRType: valArr[3],
258		FirstSeen: int(o.first_seen),
259		LastSeen: int(o.last_seen),
260		})
261		}
262
263		return outs, nil
264		}
265
266		//export cgoObsDump
267		func cgoObsDump(o *C.Observation) {
268		valArr := strings.Split(C.GoString(o.key), keySepChar)
269		myObs := observation.Observation{
270		SensorID: valArr[2],
271		Count: int(o.count),
272		RData: valArr[4],
273		RRName: valArr[1],
274		RRType: valArr[3],
275		FirstSeen: int(o.first_seen),
276		LastSeen: int(o.last_seen),
277		}
278		js, err := json.Marshal(myObs)
279		if err == nil {
280		fmt.Println(string(js))
281		}
282		}
283
284		// Dump prints all aggregated observations in the database to stdout, in JSON format.
285		func (db *RocksDB) Dump() error {
286		e := C.error_new()
287		defer C.error_delete(e)
288
289		C.obsdb_dump(db.db, e)
290
291		if C.error_is_set(e) {
292		return fmt.Errorf("%s", C.GoString(C.error_get(e)))
293		}
294
295		return nil
296		}
297
298		// TotalCount returns the overall number of observations across all sensors.
299		func (db *RocksDB) TotalCount() (int, error) {
300		var val int
301		val = int(C.obsdb_num_keys(db.db))
302		return val, nil
303		}
304
305		// Shutdown closes the database connection, leaving the database unable to
306		// process both reads and writes.
307		func (db *RocksDB) Shutdown() {
308		close(db.stopChan)
309		C.obsdb_close(db.db)
310		log.Info("database closed")
311		}

-190

~~db/db_rocksdb_test.go~~ less more

0		package db
1
2		import (
3		"io/ioutil"
4		"os"
5		"syscall"
6		"testing"
7		"time"
8
9		"github.com/DCSO/balboa/observation"
10		)
11
12		func createRocksTmpDB(t testing.T) (RocksDB, string) {
13		dbdir, err := ioutil.TempDir("", "example")
14		if err != nil {
15		t.Fatal(err)
16		}
17		db, err := MakeRocksDB(dbdir, 8000000)
18		if err != nil {
19		t.Fatal(err)
20		}
21		if db == nil {
22		t.Fatal("no db created")
23		}
24		return db, dbdir
25		}
26
27		func TestRocksCreate(t *testing.T) {
28		db, dbdir := createRocksTmpDB(t)
29		defer os.RemoveAll(dbdir)
30		defer db.Shutdown()
31		}
32
33		func TestRocksCreateFail(t *testing.T) {
34		// skip this test if run as root
35		if syscall.Getuid() == 0 {
36		t.Skip()
37		}
38		_, err := MakeRocksDB("/nonexistent", 8000000)
39		if err == nil {
40		t.Fatal(err)
41		}
42		}
43
44		func TestRocksSimpleStore(t *testing.T) {
45		db, dbdir := createRocksTmpDB(t)
46		defer os.RemoveAll(dbdir)
47		defer db.Shutdown()
48
49		a := db.AddObservation(observation.InputObservation{
50		Rrname: "foo.bar",
51		Rrtype: "A",
52		SensorID: "deadcafe",
53		Rdata: "12.34.56.78",
54		TimestampEnd: time.Now(),
55		TimestampStart: time.Now(),
56		Count: 1,
57		})
58		if a.RData != "12.34.56.78" \|\| a.RRName != "foo.bar" \|\| a.RRType != "A" \|\|
59		a.Count != 1 {
60		t.Fatal("invalid return")
61		}
62		a = db.AddObservation(observation.InputObservation{
63		Rrname: "foo.bar",
64		Rrtype: "MX",
65		SensorID: "deadcafe",
66		Rdata: "12.34.56.77",
67		TimestampEnd: time.Now(),
68		TimestampStart: time.Now(),
69		Count: 1,
70		})
71		if a.RData != "12.34.56.77" \|\| a.RRName != "foo.bar" \|\| a.RRType != "MX" \|\|
72		a.Count != 1 {
73		t.Fatal("invalid return")
74		}
75		a = db.AddObservation(observation.InputObservation{
76		Rrname: "foo.bar",
77		Rrtype: "NS",
78		SensorID: "deadcafe",
79		Rdata: "12.34.56.79",
80		TimestampEnd: time.Now(),
81		TimestampStart: time.Now(),
82		Count: 1,
83		})
84		if a.RData != "12.34.56.79" \|\| a.RRName != "foo.bar" \|\| a.RRType != "NS" \|\|
85		a.Count != 1 {
86		t.Fatal("invalid return")
87		}
88
89		str := "foo.bar"
90		obs, err := db.Search(nil, &str, nil, nil)
91		if err != nil {
92		t.Fatal(err)
93		}
94		if len(obs) != 3 {
95		t.Fatalf("wrong number of results: %d", len(obs))
96		}
97
98		a = db.AddObservation(observation.InputObservation{
99		Rrname: "foo.bar",
100		Rrtype: "NS",
101		SensorID: "deadcafe",
102		Rdata: "12.34.56.79",
103		TimestampEnd: time.Now().Add(77 * time.Second),
104		TimestampStart: time.Now().Add(77 * time.Second),
105		Count: 4,
106		})
107		if a.RData != "12.34.56.79" \|\| a.RRName != "foo.bar" \|\| a.RRType != "NS" \|\|
108		a.Count != 5 {
109		t.Fatalf("invalid return %v", a)
110		}
111
112		obs, err = db.Search(nil, &str, nil, nil)
113		if err != nil {
114		t.Fatal(err)
115		}
116		if len(obs) != 3 {
117		t.Fatalf("wrong number of results: %d", len(obs))
118		}
119		}
120
121		func TestRocksSimpleConsume(t *testing.T) {
122		db, dbdir := createRocksTmpDB(t)
123		defer os.RemoveAll(dbdir)
124		defer db.Shutdown()
125
126		inChan := make(chan observation.InputObservation)
127		stopChan := make(chan bool)
128
129		go db.ConsumeFeed(inChan)
130
131		inChan <- observation.InputObservation{
132		Rrname: "foo.bar",
133		Rrtype: "A",
134		SensorID: "deadcafe",
135		Rdata: "12.34.56.78",
136		TimestampEnd: time.Now(),
137		TimestampStart: time.Now(),
138		Count: 1,
139		}
140		inChan <- observation.InputObservation{
141		Rrname: "foo.bar",
142		Rrtype: "MX",
143		SensorID: "deadcafe",
144		Rdata: "12.34.56.77",
145		TimestampEnd: time.Now(),
146		TimestampStart: time.Now(),
147		Count: 1,
148		}
149		for i := 0; i < 10000; i++ {
150		inChan <- observation.InputObservation{
151		Rrname: "foo.bar",
152		Rrtype: "NS",
153		SensorID: "deadcafe",
154		Rdata: "12.34.56.79",
155		TimestampEnd: time.Now(),
156		TimestampStart: time.Now(),
157		Count: 2,
158		}
159		}
160		close(stopChan)
161
162		str := "foo.bar"
163		obs, err := db.Search(nil, &str, nil, nil)
164		if err != nil {
165		t.Fatal(err)
166		}
167		if len(obs) != 3 {
168		t.Fatalf("wrong number of results: %d", len(obs))
169		}
170
171		str = "12.34.56.79"
172		obs, err = db.Search(&str, nil, nil, nil)
173		if err != nil {
174		t.Fatal(err)
175		}
176		if len(obs) != 1 {
177		t.Fatalf("wrong number of results: %d", len(obs))
178		}
179
180		str = "12.34.56.79"
181		obs, err = db.Search(&str, nil, nil, &str)
182		if err != nil {
183		t.Fatal(err)
184		}
185		if len(obs) != 0 {
186		t.Fatalf("wrong number of results: %d", len(obs))
187		}
188
189		}

-699

~~db/obs_rocksdb.c~~ less more

0		/*
1		balboa
2		Copyright (c) 2018, DCSO GmbH
3		*/
4
5		#include "obs_rocksdb.h"
6
7		#include <stdlib.h>
8		#include <stdio.h>
9		#include <string.h>
10		#include "rocksdb/c.h"
11		#include "tpl.h"
12
13		extern void cgoLogInfo(const char*);
14		extern void cgoLogDebug(const char*);
15		extern void cgoObsDump(Observation*);
16
17		#define OBS_SET_STEP_SIZE 1000
18
19		struct Error {
20		char *msg;
21		};
22
23		Error* error_new()
24		{
25		Error *e = calloc(1, sizeof(Error));
26		if (!e) {
27		return NULL;
28		}
29		return e;
30		}
31
32		const char* error_get(Error *e)
33		{
34		if (!e)
35		return NULL;
36		return e->msg;
37		}
38
39		static void error_set(Error e, const char msg)
40		{
41		if (!e)
42		return;
43		if (!msg)
44		return;
45		if (e->msg)
46		free(e->msg);
47		e->msg = strdup(msg);
48		}
49
50		void error_unset(Error *e)
51		{
52		if (!e)
53		return;
54		if (e->msg)
55		free(e->msg);
56		e->msg = NULL;
57		}
58
59		bool error_is_set(Error *e)
60		{
61		if (!e)
62		return NULL;
63		return (e->msg != NULL);
64		}
65
66		void error_delete(Error *e)
67		{
68		if (!e)
69		return;
70		if (e->msg)
71		free(e->msg);
72		free(e);
73		}
74
75
76		struct ObsSet {
77		unsigned long size, used;
78		Observation **os;
79		};
80
81		static ObsSet* obs_set_create(unsigned long size)
82		{
83		ObsSet *os = calloc((size_t) 1, sizeof(ObsSet));
84		if (!os)
85		return NULL;
86		os->size = size;
87		os->used = 0;
88		os->os = calloc((size_t) size, sizeof(Observation*));
89		if (os->os == NULL) {
90		free(os);
91		return NULL;
92		}
93		return os;
94		}
95
96		unsigned long obs_set_size(ObsSet *os)
97		{
98		if (!os)
99		return 0;
100		return os->used;
101		}
102
103		static void obs_set_add(ObsSet os, Observation o)
104		{
105		if (!os \|\| !os->os)
106		return;
107		if (os->used == os->size) {
108		os->size += OBS_SET_STEP_SIZE;
109		os->os = realloc(os->os, (size_t) (os->size * sizeof(Observation*)));
110		}
111		os->os[os->used++] = o;
112		}
113
114		const Observation* obs_set_get(ObsSet *os, unsigned long i)
115		{
116		if (!os)
117		return NULL;
118		if (i >= os->used)
119		return NULL;
120		return os->os[i];
121		}
122
123		void obs_set_delete(ObsSet *os)
124		{
125		if (!os)
126		return;
127		if (os->os != NULL) {
128		unsigned long i = 0;
129		for (i = 0; i < os->used; i++) {
130		if (os->os[i]->key)
131		free(os->os[i]->key);
132		if (os->os[i]->inv_key)
133		free(os->os[i]->inv_key);
134		free(os->os[i]);
135		}
136		free(os->os);
137		}
138		free(os);
139		}
140
141		struct ObsDB {
142		rocksdb_t *db;
143		rocksdb_options_t *options;
144		rocksdb_writeoptions_t *writeoptions;
145		rocksdb_readoptions_t *readoptions;
146		rocksdb_mergeoperator_t *mergeop;
147		};
148
149		#define VALUE_LENGTH (sizeof(uint32_t) + (2 * sizeof(time_t)))
150
151		#define obsdb_max(a,b) \
152		({ __typeof__ (a) _a = (a); \
153		__typeof__ (b) _b = (b); \
154		_a > _b ? _a : _b; })
155
156		#define obsdb_min(a,b) \
157		({ __typeof__ (a) _a = (a); \
158		__typeof__ (b) _b = (b); \
159		_a < _b ? _a : _b; })
160
161		static inline int obs2buf(Observation o, char buf, size_t buflen) {
162		uint32_t a,b,c;
163		int ret = 0;
164		tpl_node *tn = tpl_map("uuu", &a, &b, &c);
165		a = o->count;
166		b = o->last_seen;
167		c = o->first_seen;
168		ret = tpl_pack(tn, 0);
169		if (ret == 0) {
170		ret = tpl_dump(tn, TPL_MEM, buf, buflen);
171		}
172		tpl_free(tn);
173		return ret;
174		}
175
176		static inline int buf2obs(Observation o, const char buf, size_t buflen) {
177		uint32_t a,b,c;
178		int ret = 0;
179		tpl_node *tn = tpl_map("uuu", &a, &b, &c);
180		ret = tpl_load(tn, TPL_MEM, buf, buflen);
181		if (ret == 0) {
182		(void) tpl_unpack(tn, 0);
183		o->count = a;
184		o->last_seen = b;
185		o->first_seen = c;
186		}
187		tpl_free(tn);
188		return ret;
189		}
190
191		static char* obsdb_mergeop_full_merge(void state, const char key,
192		size_t key_length, const char* existing_value,
193		size_t existing_value_length,
194		const char* const* operands_list,
195		const size_t* operands_list_length,
196		int num_operands, unsigned char* success,
197		size_t* new_value_length)
198		{
199		Observation obs = {NULL, NULL, 0, 0, 0};
200
201		if (key_length < 1) {
202		fprintf(stderr, "full merge: key too short\n");
203		*success = (unsigned char) 0;
204		return NULL;
205		}
206		if (key[0] == 'i') {
207		/* this is an inverted index key with no meaningful value */
208		char res = malloc(1 sizeof(char));
209		*res = '\0';
210		*new_value_length = 1;
211		*success = 1;
212		return res;
213		} else if (key[0] == 'o') {
214		/* this is an observation value */
215		int i;
216		size_t buflength;
217		char *buf = NULL;
218		if (existing_value) {
219		buf2obs(&obs, existing_value, existing_value_length);
220		}
221		for (i = 0; i < num_operands; i++) {
222		Observation nobs = {NULL, NULL, 0, 0, 0};
223		buf2obs(&nobs, operands_list[i], operands_list_length[i]);
224		if (i == 0) {
225		if (!existing_value) {
226		obs.count = nobs.count;
227		obs.last_seen = nobs.last_seen;
228		obs.first_seen = nobs.first_seen;
229		} else {
230		obs.count += nobs.count;
231		obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
232		obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
233		}
234		} else {
235		obs.count += nobs.count;
236		obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
237		obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
238		}
239		}
240		obs2buf(&obs, &buf, &buflength);
241		*new_value_length = buflength;
242		*success = (unsigned char) 1;
243		return buf;
244		} else {
245		/* weird key format! */
246		fprintf(stderr, "full merge: weird key format\n");
247		*success = (unsigned char) 0;
248		return NULL;
249		}
250		}
251
252		static char* obsdb_mergeop_partial_merge(void state, const char key,
253		size_t key_length,
254		const char* const* operands_list,
255		const size_t* operands_list_length,
256		int num_operands, unsigned char* success,
257		size_t* new_value_length)
258		{
259		Observation obs = {NULL, NULL, 0, 0, 0};
260
261		if (key_length < 1) {
262		fprintf(stderr, "partial merge: key too short\n");
263		*success = (unsigned char) 0;
264		return NULL;
265		}
266		if (key[0] == 'i') {
267		/* this is an inverted index key with no meaningful value */
268		char res = malloc(1 sizeof(char));
269		*res = '\0';
270		*new_value_length = 1;
271		*success = 1;
272		return res;
273		} else if (key[0] == 'o') {
274		/* this is an observation value */
275		int i;
276		size_t buflength;
277		char *buf = NULL;
278		for (i = 0; i < num_operands; i++) {
279		Observation nobs = {NULL, NULL, 0, 0, 0};
280		buf2obs(&nobs, operands_list[i], operands_list_length[i]);
281		if (i == 0) {
282		obs.count = nobs.count;
283		obs.last_seen = nobs.last_seen;
284		obs.first_seen = nobs.first_seen;
285		} else {
286		obs.count += nobs.count;
287		obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
288		obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
289		}
290		}
291		obs2buf(&obs, &buf, &buflength);
292		*new_value_length = buflength;
293		*success = (unsigned char) 1;
294		return buf;
295		} else {
296		/* weird key format! */
297		fprintf(stderr, "partial merge: weird key format\n");
298		*success = (unsigned char) 0;
299		return NULL;
300		}
301		}
302
303		static void obsdb_mergeop_destructor(void *state)
304		{
305		return;
306		}
307
308		static const char* obsdb_mergeop_name(void *state)
309		{
310		return "observation mergeop";
311		}
312
313		static ObsDB* _obsdb_open(const char path, size_t membudget, Error e, bool readonly)
314		{
315		char *err = NULL;
316		int level_compression[5] = {
317		rocksdb_lz4_compression,
318		rocksdb_lz4_compression,
319		rocksdb_lz4_compression,
320		rocksdb_lz4_compression,
321		rocksdb_lz4_compression
322		};
323		ObsDB *db = calloc(1, sizeof(ObsDB));
324		if (db == NULL) {
325		if (e)
326		error_set(e, "could not allocate memory");
327		return NULL;
328		}
329
330		db->mergeop = rocksdb_mergeoperator_create(NULL,
331		obsdb_mergeop_destructor,
332		obsdb_mergeop_full_merge,
333		obsdb_mergeop_partial_merge,
334		NULL,
335		obsdb_mergeop_name);
336
337		db->options = rocksdb_options_create();
338		rocksdb_options_increase_parallelism(db->options, 8);
339		if (!readonly)
340		rocksdb_options_optimize_level_style_compaction(db->options, membudget);
341		rocksdb_options_set_create_if_missing(db->options, 1);
342		rocksdb_options_set_max_log_file_size(db->options, 1010241024);
343		rocksdb_options_set_keep_log_file_num(db->options, 2);
344		rocksdb_options_set_max_open_files(db->options, 300);
345		rocksdb_options_set_merge_operator(db->options, db->mergeop);
346		rocksdb_options_set_compression_per_level(db->options, level_compression, 5);
347
348		if (!readonly)
349		db->db = rocksdb_open(db->options, path, &err);
350		else
351		db->db = rocksdb_open_for_read_only(db->options, path, 0, &err);
352		if (err) {
353		if (e)
354		error_set(e, err);
355		free(err);
356		return NULL;
357		}
358
359		db->writeoptions = rocksdb_writeoptions_create();
360		db->readoptions = rocksdb_readoptions_create();
361
362		return db;
363		}
364
365		ObsDB* obsdb_open(const char path, size_t membudget, Error e) {
366		return _obsdb_open(path, membudget, e, false);
367		}
368
369		ObsDB* obsdb_open_readonly(const char path, Error e) {
370		return _obsdb_open(path, 0, e, true);
371		}
372
373		int obsdb_put(ObsDB db, Observation obs, Error *e)
374		{
375		char *err = NULL;
376		size_t buflength;
377		char *buf;
378		if (!db)
379		return -1;
380
381		(void) obs2buf(obs, &buf, &buflength);
382
383		rocksdb_merge(db->db, db->writeoptions, obs->key, strlen(obs->key),
384		buf, buflength, &err);
385		if (err) {
386		if (e)
387		error_set(e, err);
388		free(err);
389		free(buf);
390		return -1;
391
392		}
393		free(buf);
394
395		rocksdb_merge(db->db, db->writeoptions, obs->inv_key, strlen(obs->key),
396		"", 0, &err);
397		if (err) {
398		if (e)
399		error_set(e, err);
400		free(err);
401		return -1;
402		}
403
404		return 0;
405		}
406
407		int obsdb_dump(ObsDB db, Error e)
408		{
409		rocksdb_iterator_t *it;
410		if (!db)
411		return -1;
412
413		it = rocksdb_create_iterator(db->db, db->readoptions);
414		for (rocksdb_iter_seek(it, "o", 1);
415		rocksdb_iter_valid(it) != (unsigned char) 0;
416		rocksdb_iter_next(it)) {
417		size_t size = 0;
418		int ret = 0;
419		Observation *o = NULL;
420		const char rkey = NULL, val = NULL;
421
422		rkey = rocksdb_iter_key(it, &size);
423		o = calloc(1, sizeof(Observation));
424		if (!o) {
425		return -1;
426		}
427		o->key = calloc(size + 1, sizeof(char));
428		if (!o->key) {
429		free(o);
430		return -1;
431		}
432		strncpy(o->key, rkey, size);
433		o->key[size] = '\0';
434		val = rocksdb_iter_value(it, &size);
435		ret = buf2obs(o, val, size);
436		if (ret != 0) {
437		fprintf(stderr, "error\n");
438		}
439		cgoObsDump(o);
440		free(o->key);
441		free(o);
442		}
443		rocksdb_iter_destroy(it);
444
445		return 0;
446		}
447
448		ObsSet* obsdb_search(ObsDB db, const char qrdata, const char *qrrname,
449		const char qrrtype, const char qsensorID)
450		{
451		rocksdb_iterator_t *it;
452		ObsSet *os;
453		if (!db)
454		return NULL;
455
456		os = obs_set_create(OBS_SET_STEP_SIZE);
457
458		if (qrrname != NULL) {
459		char *prefix = NULL;
460		size_t prefixlen = 0;
461		if (qsensorID != NULL) {
462		prefixlen = strlen(qsensorID) + strlen(qrrname) + 4;
463		prefix = calloc(prefixlen, sizeof(char));
464		if (!prefix) {
465		obs_set_delete(os);
466		return NULL;
467		}
468		(void) snprintf(prefix, prefixlen, "o%c%s%c%s%c", 0x1f, qrrname, 0x1f, qsensorID, 0x1f);
469		} else {
470		prefixlen = strlen(qrrname) + 3;
471		prefix = calloc(prefixlen, sizeof(char));
472		if (!prefix) {
473		obs_set_delete(os);
474		return NULL;
475		}
476		(void) snprintf(prefix, prefixlen, "o%c%s%c", 0x1f, qrrname, 0x1f);
477		}
478		cgoLogDebug(prefix);
479
480		it = rocksdb_create_iterator(db->db, db->readoptions);
481		rocksdb_iter_seek(it, prefix, prefixlen);
482		for (; rocksdb_iter_valid(it) != (unsigned char) 0; rocksdb_iter_next(it)) {
483		size_t size = 0;
484		int ret = 0;
485		Observation *o = NULL;
486		const char rkey = rocksdb_iter_key(it, &size), val = NULL;
487		char rrname = NULL, sensorID = NULL, rrtype = NULL, rdata = NULL, *saveptr;
488		char *tokkey = calloc(size + 1, sizeof(char));
489		if (!tokkey) {
490		obs_set_delete(os);
491		rocksdb_iter_destroy(it);
492		free(prefix);
493		return NULL;
494		}
495
496		strncpy(tokkey, rkey, size);
497		tokkey[size] = '\0';
498		rrname = strtok_r(tokkey+2, "\x1f", &saveptr);
499		sensorID = strtok_r(NULL, "\x1f", &saveptr);
500		rrtype = strtok_r(NULL, "\x1f", &saveptr);
501		rdata = strtok_r(NULL, "\x1f", &saveptr);
502		if (rrname == NULL \|\| (strcmp(rrname, qrrname) != 0)) {
503		free(tokkey);
504		tokkey = NULL;
505		break;
506		}
507		if (sensorID == NULL \|\| (qsensorID != NULL && strcmp(qsensorID, sensorID) != 0)) {
508		free(tokkey);
509		tokkey = NULL;
510		continue;
511		}
512		if (rdata == NULL \|\| (qrdata != NULL && strcmp(qrdata, rdata) != 0)) {
513		free(tokkey);
514		tokkey = NULL;
515		continue;
516		}
517		if (rrtype == NULL \|\| (qrrtype != NULL && strcmp(qrrtype, rrtype) != 0)) {
518		free(tokkey);
519		tokkey = NULL;
520		continue;
521		}
522
523		o = calloc(1, sizeof(Observation));
524		if (!o) {
525		free(tokkey);
526		obs_set_delete(os);
527		rocksdb_iter_destroy(it);
528		free(prefix);
529		return NULL;
530		}
531		o->key = calloc(size + 1, sizeof(char));
532		if (!o->key) {
533		free(o);
534		free(tokkey);
535		obs_set_delete(os);
536		rocksdb_iter_destroy(it);
537		free(prefix);
538		return NULL;
539		}
540		strncpy(o->key, rkey, size);
541		o->key[size] = '\0';
542		val = rocksdb_iter_value(it, &size);
543		ret = buf2obs(o, val, size);
544		if (ret == 0) {
545		obs_set_add(os, o);
546		}
547		free(tokkey);
548		tokkey = NULL;
549		}
550		rocksdb_iter_destroy(it);
551		free(prefix);
552		} else {
553		char *prefix = NULL;
554		size_t prefixlen = 0;
555		if (qsensorID != NULL) {
556		prefixlen = strlen(qsensorID) + strlen(qrdata) + 4;
557		prefix = calloc(prefixlen, sizeof(char));
558		if (!prefix) {
559		obs_set_delete(os);
560		return NULL;
561		}
562		(void) snprintf(prefix, prefixlen, "i%c%s%c%s%c", 0x1f, qrdata, 0x1f, qsensorID, 0x1f);
563		} else {
564		prefixlen = strlen(qrdata) + 3;
565		prefix = calloc(prefixlen, sizeof(char));
566		if (!prefix) {
567		obs_set_delete(os);
568		return NULL;
569		}
570		(void) snprintf(prefix, prefixlen, "i%c%s%c", 0x1f, qrdata, 0x1f);
571		}
572		cgoLogDebug(prefix);
573
574		it = rocksdb_create_iterator(db->db, db->readoptions);
575		rocksdb_iter_seek(it, prefix, strlen(prefix));
576		for (; rocksdb_iter_valid(it) != (unsigned char) 0; rocksdb_iter_next(it)) {
577		size_t size = 0, fullkeylen;
578		int ret;
579		const char *rkey = rocksdb_iter_key(it, &size);
580		char *val = NULL;
581		char rrname = NULL, sensorID = NULL, rrtype = NULL, rdata = NULL, *saveptr;
582		char fullkey[4096];
583		char *err = NULL;
584		Observation *o = NULL;
585		char *tokkey = calloc(size + 1, sizeof(char));
586		if (!tokkey) {
587		obs_set_delete(os);
588		rocksdb_iter_destroy(it);
589		free(prefix);
590		return NULL;
591		}
592
593		strncpy(tokkey, rkey, size);
594		tokkey[size] = '\0';
595		rdata = strtok_r(tokkey+2, "\x1f", &saveptr);
596		sensorID = strtok_r(NULL, "\x1f", &saveptr);
597		rrname = strtok_r(NULL, "\x1f", &saveptr);
598		rrtype = strtok_r(NULL, "\x1f", &saveptr);
599		if (strcmp(rdata, qrdata) != 0) {
600		free(tokkey);
601		tokkey = NULL;
602		break;
603		}
604		cgoLogDebug(rdata);
605		if (sensorID == NULL \|\| (qsensorID != NULL && strcmp(qsensorID, sensorID) != 0)) {
606		free(tokkey);
607		tokkey = NULL;
608		continue;
609		}
610		if (rdata == NULL \|\| (qrdata != NULL && strcmp(qrdata, rdata) != 0)) {
611		free(tokkey);
612		tokkey = NULL;
613		continue;
614		}
615		if (rrtype == NULL \|\| (qrrtype != NULL && strcmp(qrrtype, rrtype) != 0)) {
616		free(tokkey);
617		tokkey = NULL;
618		continue;
619		}
620
621		(void) snprintf(fullkey, 4096, "o%c%s%c%s%c%s%c%s", 0x1f, rrname, 0x1f, sensorID, 0x1f, rrtype, 0x1f, rdata);
622		cgoLogDebug(fullkey);
623
624		fullkeylen = strlen(fullkey);
625		val = rocksdb_get(db->db, db->readoptions, fullkey, fullkeylen, &size, &err);
626		if (err != NULL) {
627		cgoLogDebug("observation not found");
628		free(tokkey);
629		tokkey = NULL;
630		continue;
631		}
632
633		o = calloc(1, sizeof(Observation));
634		if (!o) {
635		free(tokkey);
636		obs_set_delete(os);
637		rocksdb_iter_destroy(it);
638		free(prefix);
639		free(val);
640		return NULL;
641		}
642		o->key = calloc(fullkeylen + 1, sizeof(char));
643		if (!o->key) {
644		free(o);
645		free(tokkey);
646		obs_set_delete(os);
647		rocksdb_iter_destroy(it);
648		free(prefix);
649		free(val);
650		return NULL;
651		}
652		strncpy(o->key, fullkey, fullkeylen);
653		o->key[fullkeylen] = '\0';
654		ret = buf2obs(o, val, size);
655		if (ret == 0) {
656		obs_set_add(os, o);
657		}
658		free(tokkey);
659		free(val);
660		tokkey = NULL;
661		}
662		rocksdb_iter_destroy(it);
663		free(prefix);
664		}
665
666		return os;
667		}
668
669		unsigned long obsdb_num_keys(ObsDB *db)
670		{
671		const char *val;
672		if (!db)
673		return 0;
674		val = rocksdb_property_value(db->db, "rocksdb.estimate-num-keys");
675		if (!val) {
676		return 0;
677		} else {
678		unsigned long v;
679		int ret;
680		ret = sscanf(val, "%lu", &v);
681		if (ret != 1) {
682		return 0;
683		} else {
684		return v;
685		}
686		}
687		}
688
689		void obsdb_close(ObsDB *db)
690		{
691		if (!db)
692		return;
693		rocksdb_mergeoperator_destroy(db->mergeop);
694		rocksdb_writeoptions_destroy(db->writeoptions);
695		rocksdb_readoptions_destroy(db->readoptions);
696		// rocksdb_options_destroy(db->options);
697		rocksdb_close(db->db);
698		}

-46

~~db/obs_rocksdb.h~~ less more

0		/*
1		balboa
2		Copyright (c) 2018, DCSO GmbH
3		*/
4
5		#ifndef OBS_ROCKSDB_H
6		#define OBS_ROCKSDB_H
7
8		#include <stdbool.h>
9		#include <stdint.h>
10		#include <time.h>
11
12		typedef struct Error Error;
13
14		Error* error_new();
15		const char* error_get(Error*);
16		bool error_is_set(Error*);
17		void error_delete(Error*);
18
19		typedef struct {
20		char *key,
21		*inv_key;
22		uint32_t count,
23		last_seen,
24		first_seen;
25		} Observation;
26
27		typedef struct ObsSet ObsSet;
28
29		unsigned long obs_set_size(ObsSet*);
30		const Observation* obs_set_get(ObsSet*, unsigned long);
31		void obs_set_delete(ObsSet*);
32
33		typedef struct ObsDB ObsDB;
34
35		ObsDB* obsdb_open(const char path, size_t membudget, Error);
36		ObsDB* obsdb_open_readonly(const char path, Error);
37		int obsdb_put(ObsDB db, Observation obs, Error*);
38		ObsSet* obsdb_search(ObsDB db, const char qrdata, const char *qrrname,
39		const char qrrtype, const char qsensorID);
40		int obsdb_dump(ObsDB db, Error e) ;
41
42		unsigned long obsdb_num_keys(ObsDB*);
43		void obsdb_close(ObsDB*);
44
45		#endif⏎

+11

-11

feeder/feeder_amqp.go less more

166	166	},
167	167	)
168	168	if err != nil {
169		return fmt.Errorf("Queue Declare: %s", err)
	169	return fmt.Errorf("error queue declare: %s", err)
170	170	}
171	171	log.Debugf("declared Queue (%q %d messages, %d consumers), binding to Exchange (key %q)",
172	172	queue.Name(), queue.Messages(), queue.Consumers(), key)

181	181	"noWait": false,
182	182	},
183	183	); err != nil {
184		return fmt.Errorf("Queue Bind: %s", err)
185		}
186		}
187
188		log.Debugf("Queue bound to Exchange, starting Consume (consumer tag %q)", c.tag)
	184	return fmt.Errorf("error queue bind: %s", err)
	185	}
	186	}
	187
	188	log.Debugf("queue bound to Exchange, starting Consume (consumer tag %q)", c.tag)
189	189	c.deliveries, err = c.channel.Consume(
190	190	queue.Name(),
191	191	c.tag,

197	197	},
198	198	)
199	199	if err != nil {
200		return fmt.Errorf("Queue Consume: %s", err)
201		}
202
203		log.Debugf("Consumer established connection to %s", s.URL)
	200	return fmt.Errorf("error queue consume: %s", err)
	201	}
	202
	203	log.Debugf("consumer established connection to %s", s.URL)
204	204	s.stop = make(chan bool)
205	205	c.ErrorChan = make(chan wabbit.Error)
206	206

239	239	func (c *Consumer) Shutdown() error {
240	240	// will close() the deliveries channel
241	241	if err := c.channel.Close(); err != nil {
242		return fmt.Errorf("Channel close failed: %s", err)
	242	return fmt.Errorf("channel close failed: %s", err)
243	243	}
244	244	if err := c.conn.Close(); err != nil {
245	245	return fmt.Errorf("AMQP connection close error: %s", err)

-2

format/format_fever.go less more

13	13
14	14	type rdata struct {
15	15	AnsweringHost string `json:"answering_host"`
16		Count int `json:"count"`
	16	Count uint `json:"count"`
17	17	Rcode string `json:"rcode"`
18	18	Rdata string `json:"rdata"`
19	19	Rrtype string `json:"rrtype"`

29	29	}
30	30
31	31	// MakeFeverAggregateInputObservations is a MakeObservationFunc that accepts
32		// input in suristasher/FEVER's JSON format.
	32	// input in FEVER's JSON format.
33	33	func MakeFeverAggregateInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
34	34	var in inputJSONstruct
35	35	var i int64

-1

format/format_gamelinux.go less more

50	50	continue
51	51	}
52	52	o := observation.InputObservation{
53		Count: count,
	53	Count: uint(count),
54	54	Rdata: strings.TrimRight(rdata, "."),
55	55	Rrname: strings.TrimRight(rrname, "."),
56	56	Rrtype: rrtype,

-8

observation/input_observation.go less more

9	9	// InputObservation is a minimal, small observation structure to be used as
10	10	// the minimal common input type for all consumers.
11	11	type InputObservation struct {
12		Count int
13		Rcode string
14		Rdata string
15		Rrtype string
16		Rrname string
17		SensorID string
18		TimestampEnd time.Time
19		TimestampStart time.Time
	12	Count uint `codec:"C"`
	13	Rcode string `codec:"-"`
	14	Rdata string `codec:"D"`
	15	Rrtype string `codec:"T"`
	16	Rrname string `codec:"N"`
	17	SensorID string `codec:"I"`
	18	TimestampEnd time.Time `codec:"L"`
	19	TimestampStart time.Time `codec:"F"`
20	20	}
21	21
22	22	// InChan is the global input channel delivering InputObservations from

-8

observation/observation.go less more

4	4
5	5	import (
6	6	uuid "github.com/satori/go.uuid"
	7	"time"
7	8	)
8	9
9	10	// Observation represents a DNS answer, potentially repeated, observed on a
10	11	// given sensor stating a specific RR set.
11	12	type Observation struct {
12		ID uuid.UUID `json:"-"`
13		Count int `json:"count"`
14		FirstSeen int `json:"time_first"`
15		LastSeen int `json:"time_last"`
16		RRType string `json:"rrtype"`
17		RRName string `json:"rrname"`
18		RData string `json:"rdata"`
19		SensorID string `json:"sensor_id"`
	13	ID uuid.UUID `json:"-" codec:"-"`
	14	Count uint `json:"count" codec:"C"`
	15	FirstSeen time.Time `json:"time_first" codec:"F"`
	16	LastSeen time.Time `json:"time_last" codec:"L"`
	17	RRType string `json:"rrtype" codec:"T"`
	18	RRName string `json:"rrname" codec:"N"`
	19	RData string `json:"rdata" codec:"D"`
	20	SensorID string `json:"sensor_id" codec:"I"`
20	21	}

+46

-55

query/query_graphql.go less more

69	69	TXT
70	70	URI
71	71	}
72
73		# A single observation, unique for the combination of sensor, rrname,
	72
	73	# A single observation, unique for the combination of sensor, rrname,
74	74	# rdata and rrtype. Corresponds, roughly, to a pDNS COF item, but with
75	75	# additional Aliases (linked via IP in A/AAAA records).
76	76	type Entry {
77	77	# The number of observed occurrences of this observation.
78	78	count: Int!
79
	79
80	80	# The RRName seen in this observation.
81	81	rrname: String!
82	82

103	103
104	104	# Entries referencing the same IP (for A/AAAA) observed on the same
105	105	# sensor.
106		aliases: [LeafEntry]
	106	aliases(limit: Int = 1000): [LeafEntry]
107	107	}
108	108
109	109	# A single observation, unique for the combination of sensor, rrname,

168	168	# Number of concurrent goroutines in the server instance.
169	169	num_goroutines: Int!
170	170	}
171
	171
172	172	type Query {
173	173	# Returns a set of observations satisfying the given query parameters.
174	174	# Providing rdata, rrname, rrtype and/or sensor_id will restrict the
175	175	# results to the set of observations that match all of the given
176	176	# constraints.
177		entries(rdata: String, rrname: String, rrtype: RRType, sensor_id: String): [Entry]
178
	177	entries(rdata: String, rrname: String, rrtype: RRType, sensor_id: String, limit: Int = 1000): [Entry]
	178
179	179	# Returns some runtime values describing the current state of the database.
180	180	stats(): Stats
181	181	}
182	182
183		type Mutation {
184		announceObservation(observation: EntryInput!): Entry!
185		}
186
	183	#type Mutation {
	184	# announceObservation(observation: EntryInput!): Entry!
	185	#}
	186
187	187	schema {
188	188	query: Query
189		mutation: Mutation
	189	#mutation: Mutation
190	190	}`
191	191	)
192	192

199	199
200	200	// Resolver is just used to bundle top level methods.
201	201	type Resolver struct{}
202
203		type entryInputArgs struct {
204		Count int
205		FirstSeen int
206		LastSeen int
207		RRType string
208		RRName string
209		RData string
210		SensorID string
211		}
212	202
213	203	// Entries returns a collection of Entry resolvers, given parameters such as
214	204	// Rdata, RRname, RRtype and sensor ID.

217	207	Rrname *string
218	208	Rrtype *string
219	209	SensorID *string
	210	Limit int32
220	211	}) ([]EntryResolver, error) {
221	212	startTime := time.Now()
222	213	defer func() {

250	241	Message: "at least one of the 'rdata' or 'rrname' parameters is required",
251	242	}
252	243	}
253		results, err := db.ObservationDB.Search(args.Rdata, args.Rrname, args.Rrtype, args.SensorID)
	244	results, err := db.ObservationDB.Search(args.Rdata, args.Rrname, args.Rrtype, args.SensorID, int(args.Limit))
254	245	if err != nil {
255	246	return nil, err
256	247	}

265	256
266	257	// AnnounceObservation is a mutation that adds a single new observation
267	258	// to the database.
268		func (r *Resolver) AnnounceObservation(args struct {
269		Observation struct {
270		Count int32
271		TimeFirst int32
272		TimeLast int32
273		RRType string
274		RRName string
275		RData string
276		SensorID string
277		}
278		}) *EntryResolver {
279		inObs := observation.InputObservation{
280		Count: int(args.Observation.Count),
281		TimestampStart: time.Unix(int64(args.Observation.TimeFirst), 0),
282		TimestampEnd: time.Unix(int64(args.Observation.TimeLast), 0),
283		Rrname: args.Observation.RRName,
284		Rrtype: args.Observation.RRType,
285		Rdata: args.Observation.RData,
286		SensorID: args.Observation.SensorID,
287		}
288		resObs := db.ObservationDB.AddObservation(inObs)
289		return &EntryResolver{
290		entry: resObs,
291		}
292		}
	259	//func (r *Resolver) AnnounceObservation(args struct {
	260	// Observation struct {
	261	// Count int32
	262	// TimeFirst int32
	263	// TimeLast int32
	264	// RRType string
	265	// RRName string
	266	// RData string
	267	// SensorID string
	268	// }
	269	//}) *EntryResolver {
	270	// inObs := observation.InputObservation{
	271	// Count: int(args.Observation.Count),
	272	// TimestampStart: time.Unix(int64(args.Observation.TimeFirst), 0),
	273	// TimestampEnd: time.Unix(int64(args.Observation.TimeLast), 0),
	274	// Rrname: args.Observation.RRName,
	275	// Rrtype: args.Observation.RRType,
	276	// Rdata: args.Observation.RData,
	277	// SensorID: args.Observation.SensorID,
	278	// }
	279	// resObs := db.ObservationDB.AddObservation(inObs)
	280	// return &EntryResolver{
	281	// entry: resObs,
	282	// }
	283	//}
293	284
294	285	// Stats returns a Stats resolver.
295	286	func (r Resolver) Stats() (StatsResolver, error) {

298	289
299	290	// StatsResolver is a resolver for the Stats type.
300	291	type StatsResolver struct {
301		totalCount int32
	292	//totalCount int32
302	293	}
303	294
304	295	// TotalCount returns the total number of keys in the database.

349	340
350	341	// TimeFirst returns the first seen timestamp of the corresponding entry.
351	342	func (r *EntryResolver) TimeFirst() int32 {
352		return int32(r.entry.FirstSeen)
	343	return int32(r.entry.FirstSeen.Unix())
353	344	}
354	345
355	346	// TimeFirstRFC3339 returns first seen time, as RFC 3339 string, of the corresponding entry.
356	347	func (r *EntryResolver) TimeFirstRFC3339() string {
357		return time.Unix(int64(r.entry.FirstSeen), 0).Format(time.RFC3339)
	348	return r.entry.FirstSeen.Format(time.RFC3339)
358	349	}
359	350
360	351	// TimeLast returns the last seen timestamp of the corresponding entry.
361	352	func (r *EntryResolver) TimeLast() int32 {
362		return int32(r.entry.LastSeen)
	353	return int32(r.entry.LastSeen.Unix())
363	354	}
364	355
365	356	// TimeLastRFC3339 returns last seen time, as RFC 3339 string, of the corresponding entry.
366	357	func (r *EntryResolver) TimeLastRFC3339() string {
367		return time.Unix(int64(r.entry.LastSeen), 0).Format(time.RFC3339)
	358	return r.entry.LastSeen.Format(time.RFC3339)
368	359	}
369	360
370	361	// SensorID returns the sensor ID field of the corresponding entry.

374	365
375	366	// Aliases returns resolvers for Entries with the same IPs in Rdata (for
376	367	// A/AAAA type entries).
377		func (r EntryResolver) Aliases() []*EntryResolver {
	368	func (r EntryResolver) Aliases(args struct{ Limit int32 }) []*EntryResolver {
378	369	l := make([]*EntryResolver, 0)
379	370	if !(r.entry.RRType == "A" \|\| r.entry.RRType == "AAAA") {
380	371	return nil
381	372	}
382		results, err := db.ObservationDB.Search(&r.entry.RData, nil, nil, &r.entry.SensorID)
	373	results, err := db.ObservationDB.Search(&r.entry.RData, nil, nil, &r.entry.SensorID, int(args.Limit))
383	374	if err != nil {
384	375	return nil
385	376	}

+10

-0

shell.nix less more

	0
	1	{ pkgs ? import <nixpkgs> {} }: let
	2	#rocksdb = pkgs.callPackage ./lib/rocksdb.nix {};
	3	in
	4	pkgs.mkShell {
	5	shellHook = ''
	6	export GOPATH=${toString ./.}/go
	7	'';
	8	buildInputs = with pkgs; [ git go go2nix ];
	9	}