Codebase list balboa / 5396503
New upstream version 1.0 Sascha Steinbiss 5 years ago
48 changed file(s) with 5682 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
+22
-0
.circleci/config.yml less more
0 # Golang CircleCI 2.0 configuration file
1 #
2 # Check https://circleci.com/docs/2.0/language-go/ for more details
3 version: 2
4 jobs:
5 build:
6 docker:
7 - image: circleci/golang:1.10-stretch
8
9 working_directory: /go/src/github.com/DCSO/balboa
10 steps:
11 - checkout
12 - run:
13 name: Add stretch-backports
14 command: 'echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee -a /etc/apt/sources.list.d/backports.list'
15 - run:
16 name: Install apt dependencies
17 command: 'sudo apt-get update && sudo apt-get -t stretch-backports install librocksdb-dev libtpl-dev -y'
18
19 # specify any bash command here prefixed with `run: `
20 - run: go get -v -t -d ./...
21 - run: go test -v ./...
+5
-0
.travis.yml less more
0 language: go
1 sudo: false
2 script:
3 - go vet ./...
4 - go test -v ./...
+27
-0
LICENSE less more
0 Copyright (c) 2018, DCSO Deutsche Cyber-Sicherheitsorganisation GmbH
1 All rights reserved.
2
3 Redistribution and use in source and binary forms, with or without
4 modification, are permitted provided that the following conditions are met:
5
6 * Redistributions of source code must retain the above copyright notice, this
7 list of conditions and the following disclaimer.
8
9 * Redistributions in binary form must reproduce the above copyright notice,
10 this list of conditions and the following disclaimer in the documentation
11 and/or other materials provided with the distribution.
12
13 * Neither the name of the DCSO Deutsche Cyber-Sicherheitsorganisation GmbH
14 nor the names of its contributors may be used to endorse or promote products
15 derived from this software without specific prior written permission.
16
17 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
18 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
19 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
20 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
21 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
22 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
23 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
24 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
25 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+275
-0
README.md less more
0 # balboa [![CircleCI](https://circleci.com/gh/DCSO/balboa.svg?style=svg)](https://circleci.com/gh/DCSO/balboa)
1
2 balboa is the BAsic Little Book Of Answers. It consumes and indexes observations from [passive DNS](https://www.farsightsecurity.com/technical/passive-dns/) collection, providing a [GraphQL](https://graphql.org/) interface to access the aggregated contents of the observations database. We built balboa to handle passive DNS data aggregated from metadata gathered by [Suricata](https://suricata-ids.org).
3
4 The API should be suitable for integration into existing multi-source observable integration frameworks. It is possible to produce results in a [Common Output Format](https://datatracker.ietf.org/doc/draft-dulaunoy-dnsop-passive-dns-cof/) compatible schema using the GraphQL API. In fact, the GraphQL schema is modelled after the COF fields.
5
6 The balboa software...
7
8 - is fast for queries and input/updates
9 - implements persistent, compressed storage of observations
10 - as local storage
11 - in a [Cassandra](https://cassandra.apache.org) cluster (experimental)
12 - supports tracking and specifically querying multiple sensors
13 - makes use of multi-core systems
14 - can accept input from multiple sources simultaneously
15 - HTTP (POST)
16 - AMQP
17 - GraphQL
18 - Unix socket
19 - accepts various (text-based) input formats
20 - JSON-based
21 - [FEVER](https://github.com/DCSO/fever)
22 - [gopassivedns](https://github.com/Phillipmartin/gopassivedns)
23 - [Packetbeat](https://www.elastic.co/guide/en/beats/packetbeat/master/packetbeat-dns-options.html) (via Logstash)
24 - [Suricata EVE DNS v1 and v2](http://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html#event-type-dns)
25 - flat file
26 - Edward Fjellskål's [PassiveDNS](https://github.com/gamelinux/passivedns) tabular format (default order `-f SMcsCQTAtn`)
27
28 ## Building and Installation
29
30 ```
31 $ go get github.com/DCSO/balboa
32 ```
33
34 Dependencies:
35
36 - Go 1.7 or later
37 - [RocksDB](https://rocksdb.org/) 5.0 or later (shared lib, with LZ4 support)
38 - [tpl](https://troydhanson.github.io/tpl/index.html) (shared lib)
39
40 On Debian (testing and stretch-backports), one can satisfy these dependencies with:
41
42 ```
43 apt install librocksdb-dev libtpl-dev
44 ```
45
46 ## Usage
47
48 ### Configuring feeders
49
50 Feeders are used to get observations into the database. They run concurrently and process inputs in the background, making results accessible via the query interface as soon as the resulting upsert transactions have been completed in the database. What feeders are to be created is defined in a YAML configuration file (to be passed via the `-f` parameter to `balboa serve`). Example:
51
52 ```yaml
53 feeder:
54 - name: AMQP Input
55 type: amqp
56 url: amqp://guest:guest@localhost:5672
57 exchange: [ tdh.pdns ]
58 input_format: fever_aggregate
59 - name: HTTP Input
60 type: http
61 listen_host: 127.0.0.1
62 listen_port: 8081
63 input_format: fever_aggregate
64 - name: Socket Input
65 type: socket
66 path: /tmp/balboa.sock
67 input_format: gopassivedns
68 ```
69
70 A balboa instance given this feeder configuration would support the following input options:
71
72 - JSON in FEVER's aggregate format delivered via AMQP from a temporary queue attached to the exchange `tdh.pdns` on `localhost` port 5762, authenticated with user `guest` and password `guest`
73 - JSON in FEVER's aggregate format parsed from HTTP POST requests on port 8081 on the local system
74 - JSON in gopassivedns's format, fed into the UNIX socket `/tmp/balboa.sock` created by balboa
75
76 All of these feeders accept input simultaneously, there is no distinction made as to where an observation has come from. It is possible to specify multiple feeders of the same type but with different settings as long as their `name`s are unique.
77
78 ### Configuring the database backend
79
80 Multiple database backends are supported to store pDNS observations persistently. The one to use in a particular instance can be configured separately in another YAML file (to be passed via the `-d` parameter to `balboa serve`). For example, to use the RocksDB backend storing data in `/tmp/balboa`, one would specify:
81
82 ```yaml
83 database:
84 name: Local RocksDB
85 type: rocksdb
86 db_path: /tmp/balboa
87 ```
88
89 To alternatively, for instance, send all data to a Cassandra cluster, use the following configuration:
90
91 ```yaml
92 database:
93 name: Cassandra cluster
94 type: cassandra
95 hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]
96 username: cassandra
97 password: cassandra
98 ```
99
100 which would use the specified nodes to access the cluster. Only one database can be configured at a time.
101
102 ### Running the server and consuming input
103
104 All interaction with the service on the command line takes place via the `balboa` executable. The server can be started using:
105
106 ```
107 $ balboa serve -l ''
108 INFO[0000] Local RocksDB: memory budget empty, using default of 128MB
109 INFO[0000] starting database Local RocksDB
110 INFO[0000] opening database...
111 INFO[0000] database ready
112 INFO[0000] starting feeder AMQPInput2
113 INFO[0000] starting feeder HTTP Input
114 INFO[0000] accepting submissions on port 8081
115 INFO[0000] starting feeder Socket Input
116 INFO[0000] starting feeder Suricata Socket Input
117 INFO[0000] serving GraphQL on port 8080
118 ```
119
120 Depending on the size of the existing database, the start-up time ("opening database...") might be rather long if there is a lot of log to go through.
121
122 After startup, the feeders are free to be used for data ingest. For example, one might do some of the following to test data consumption (assuming the feeders above are used):
123
124 - for AMQP:
125
126 ```
127 $ scripts/mkjson.py | rabbitmqadmin publish routing_key="" exchange=tdh.pdns
128 ```
129
130 - for HTTP:
131 ```
132 $ scripts/mkjson.py | curl -d@- -qs --header "X-Sensor-ID: abcde" http://localhost:8081/submit
133 ```
134
135 - for socket:
136 ```
137 $ sudo gopassivedns -dev eth0 | socat /tmp/balboa.sock STDIN
138 ```
139
140 Besides these asynchronous updates, it is always possible to use the `announceObservation` mutation in the GraphQL interface to explicitly add an observation, returning the updated data entry immediately:
141
142 ```graphql
143 mutation {
144 announceObservation(observation: {
145 rrname: "test.foobar.de",
146 rrtype: A,
147 rdata: "1.2.3.4",
148 time_first: 1531943211,
149 time_last: 1531949570,
150 sensor_id: "abcde",
151 count: 3
152 }) {
153 count
154 }
155 }
156 ```
157
158 This request would synchronously add a new observation with the given input data to the database and then return the new, updated `count` value.
159
160 ### Querying the server
161
162 The intended main interface for interacting with the server is via GraphQL. For example, the query
163
164 ```graphql
165 query {
166 entries(rrname: "test.foobar.de", sensor_id: "abcde") {
167 rrname
168 rrtype
169 rdata
170 time_first
171 time_last
172 sensor_id
173 count
174 }
175 }
176 ```
177
178 would return something like
179
180 ```json
181 {
182 "data": {
183 "entries": [
184 {
185 "rrname": "test.foobar.de",
186 "rrtype": "A",
187 "rdata": "1.2.3.4",
188 "time_first": 1531943211,
189 "time_last": 1531949570,
190 "sensor_id": "abcde",
191 "count": 3
192 }
193 ]
194 }
195 }
196 ```
197
198 This also works with `rdata` as the query parameter, but at least one of `rrname` or `rdata` must be stated. If there is no `sensor_id` parameter, then all results will be returned regardless of where the DNS answer was observed. Use the `time_first_rfc3339` and `time_last_rfc3339` instead of `time_first` and `time_last`, respectively, to get human-readable timestamps.
199
200 ### Aliases
201
202 Sometimes it is interesting to ask for all the domain names that resolve to the same IP address. For this reason, the GraphQL API supports a virtual `aliases` field that returns all Entries with RRType `A` or `AAAA` that share the same address in the Rdata field.
203
204 Example:
205
206 ```graphql
207 {
208 entries(rrname: "heise.de", rrtype: A) {
209 rrname
210 rdata
211 rrtype
212 time_first_rfc3339
213 time_last_rfc3339
214 aliases {
215 rrname
216 time_first_rfc3339
217 time_last_rfc3339
218 }
219 }
220 }
221 ```
222
223 ```json
224 {
225 "data": {
226 "entries": [
227 {
228 "rrname": "heise.de",
229 "rdata": "193.99.144.80",
230 "rrtype": "A",
231 "time_first_rfc3339": "2018-07-10T08:05:45Z",
232 "time_last_rfc3339": "2018-10-18T09:24:38Z",
233 "aliases": [
234 {
235 "rrname": "ct.de"
236 },
237 {
238 "rrname": "ix.de"
239 },
240 {
241 "rrname": "redirector.heise.de"
242 },
243 {
244 "rrname": "www.ix.de"
245 }
246 ]
247 }
248 ]
249 }
250 }
251 ```
252
253 ### Bulk queries
254
255 There is also a shortcut tool to make 'bulk' querying easier. For example, to get all the information on the hosts in range 1.2.0.0/16 as observed by sensor `abcde`, one can use:
256
257 ```
258 $ balboa query --sensor abcde 1.2.0.0/16
259 {"count":6,"time_first":1531943211,"time_last":1531949570,"rrtype":"A","rrname":"test.foobar.de","rdata":"1.2.3.4","sensor_id":"abcde"}
260 {"count":1,"time_first":1531943215,"time_last":1531949530,"rrtype":"A","rrname":"baz.foobar.de","rdata":"1.2.3.7","sensor_id":"abcde"}
261 ```
262 Note that this tool currently only does a lot of concurrent individual queries! To improve performance in these cases it might be worthwhile to allow for range queries on the server side as well in the future.
263
264 ### Other tools
265
266 Run `balboa` without arguments to list available subcommands and get a short description of what they do.
267
268 ## Author/Contact
269
270 Sascha Steinbiss
271
272 ## License
273
274 BSD-3-clause
+17
-0
balboa.service less more
0 [Unit]
1 Description=Basic Little Book of Answers
2 Documentation=https://github.com/DCSO/balboa
3 After=network.target
4
5 [Service]
6 SyslogIdentifier=balboa
7 EnvironmentFile=-/etc/default/balboa
8 ExecStart=/usr/bin/balboa serve $BALBOA_ARGS
9 ExecStop=/usr/bin/pkill balboa
10 PIDFile=/var/run/balboa/balboa.pid
11 LimitNOFILE=200000
12 Restart=on-failure
13 RestartSec=5
14
15 [Install]
16 WantedBy=multi-user.target
+102
-0
cmd/balboa/cmds/bench.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 "context"
7 "fmt"
8 "math/rand"
9 "sync"
10 "time"
11
12 "github.com/machinebox/graphql"
13 log "github.com/sirupsen/logrus"
14 "github.com/spf13/cobra"
15 )
16
17 // benchCmd represents the bench command
18 var benchCmd = &cobra.Command{
19 Use: "bench",
20 Short: "Run a number of queries against given endpoint",
21 Long: `This command issues a potentially large number of queries against a
22 given GraphQL endpoint, for example for benchmarking. It does this in a concurrent
23 fashion.`,
24 Run: func(cmd *cobra.Command, args []string) {
25 var err error
26 var wg sync.WaitGroup
27 rand.Seed(time.Now().Unix())
28
29 var verbose bool
30 verbose, err = cmd.Flags().GetBool("verbose")
31 if err != nil {
32 log.Fatal(err)
33 }
34 if verbose {
35 log.SetLevel(log.DebugLevel)
36 }
37
38 var url string
39 url, err = cmd.Flags().GetString("url")
40 if err != nil {
41 log.Fatal(err)
42 }
43 client := graphql.NewClient(url)
44
45 var count int
46 count, err = cmd.Flags().GetInt("count")
47 if err != nil {
48 log.Fatal(err)
49 }
50
51 ipChan := make(chan string, count)
52 for i := 0; i < 5; i++ {
53 wg.Add(1)
54 go func(mywg *sync.WaitGroup) {
55 defer mywg.Done()
56 req := graphql.NewRequest(`
57 query ($ip: String!){
58 entries(rdata: $ip) {
59 rdata
60 rrname
61 rrtype
62 time_first
63 time_last
64 count
65 sensor_id
66 }
67 }`)
68 req.Header.Set("Cache-Control", "no-cache")
69 for ip := range ipChan {
70 req.Var("ip", ip)
71 var respData observationResultSet
72 if err := client.Run(context.Background(), req, &respData); err != nil {
73 log.Fatal(err)
74 }
75 if len(respData.Entries) > 0 {
76 log.Info(respData)
77 }
78 }
79 }(&wg)
80 }
81
82 for i := int(0); i < count; i++ {
83 if i%1000 == 0 {
84 log.Info(i)
85 }
86 ip := fmt.Sprintf("%d.%d.%d.%d", rand.Intn(255), rand.Intn(255),
87 rand.Intn(255), rand.Intn(255))
88 ipChan <- ip
89 }
90 close(ipChan)
91 wg.Wait()
92 },
93 }
94
95 func init() {
96 rootCmd.AddCommand(benchCmd)
97
98 benchCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
99 benchCmd.Flags().StringP("url", "u", "http://localhost:8080/query", "URL of GraphQL interface to query")
100 benchCmd.Flags().IntP("count", "c", 10000, "number of queries")
101 }
+38
-0
cmd/balboa/cmds/makeman.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 log "github.com/sirupsen/logrus"
7 "github.com/spf13/cobra"
8 "github.com/spf13/cobra/doc"
9 )
10
11 // mmanCmd represents the makeman command
12 var mmanCmd = &cobra.Command{
13 Use: "makeman [options]",
14 Short: "Create man pages",
15 Run: func(cmd *cobra.Command, args []string) {
16 targetDir, err := cmd.Flags().GetString("dir")
17 if err != nil {
18 log.Fatal(err)
19 }
20 header := &doc.GenManHeader{}
21 err = doc.GenManTree(rootCmd, header, targetDir)
22 if err != nil {
23 log.Fatal(err)
24 }
25 for _, v := range rootCmd.Commands() {
26 err = doc.GenManTree(v, header, targetDir)
27 if err != nil {
28 log.Fatal(err)
29 }
30 }
31 },
32 }
33
34 func init() {
35 rootCmd.AddCommand(mmanCmd)
36 mmanCmd.Flags().StringP("dir", "d", ".", "target directory for man pages")
37 }
+152
-0
cmd/balboa/cmds/query.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 "context"
7 "encoding/json"
8 "fmt"
9 "net"
10 "sync"
11
12 "github.com/machinebox/graphql"
13 log "github.com/sirupsen/logrus"
14 "github.com/spf13/cobra"
15 )
16
17 type observationResultSet struct {
18 Entries []observationResult
19 }
20
21 type observationResult struct {
22 Count int `json:"count"`
23 TimeFirst int `json:"time_first"`
24 TimeLast int `json:"time_last"`
25 RRType string `json:"rrtype"`
26 RRName string `json:"rrname"`
27 RData string `json:"rdata"`
28 SensorID string `json:"sensor_id"`
29 }
30
31 func incIP(ip net.IP) {
32 for j := len(ip) - 1; j >= 0; j-- {
33 ip[j]++
34 if ip[j] > 0 {
35 break
36 }
37 }
38 }
39
40 func hosts(cidr string) ([]string, error) {
41 ip, ipnet, err := net.ParseCIDR(cidr)
42 if err != nil {
43 return nil, err
44 }
45
46 var ips []string
47 for ip := ip.Mask(ipnet.Mask); ipnet.Contains(ip); incIP(ip) {
48 ips = append(ips, ip.String())
49 }
50
51 // remove network address and broadcast address
52 if len(ips) > 2 {
53 return ips[1 : len(ips)-1], nil
54 }
55 return ips, nil
56 }
57
58 var queryCmd = &cobra.Command{
59 Use: "query [netmask]",
60 Short: "Obtain information from pDNS about IP ranges",
61 Long: `This command allows to query a balboa endpoint for information regarding
62 IPs from a given range.`,
63 Run: func(cmd *cobra.Command, args []string) {
64 var err error
65
66 if len(args) == 0 {
67 log.Fatal("needs network as argument, e.g. '192.168.0.0/24'")
68 }
69
70 var verbose bool
71 verbose, err = cmd.Flags().GetBool("verbose")
72 if err != nil {
73 log.Fatal(err)
74 }
75 if verbose {
76 log.SetLevel(log.DebugLevel)
77 }
78
79 var url string
80 url, err = cmd.Flags().GetString("url")
81 if err != nil {
82 log.Fatal(err)
83 }
84 client := graphql.NewClient(url)
85
86 var senid string
87 senid, err = cmd.Flags().GetString("sensor")
88 if err != nil {
89 log.Fatal(err)
90 }
91
92 var wg sync.WaitGroup
93 ipChan := make(chan string, 1000)
94 for i := 0; i < 5; i++ {
95 wg.Add(1)
96 go func(mywg *sync.WaitGroup) {
97 defer mywg.Done()
98 req := graphql.NewRequest(`
99 query ($ip: String!, $senid: String){
100 entries(rdata: $ip, sensor_id: $senid) {
101 rdata
102 rrname
103 rrtype
104 time_first
105 time_last
106 count
107 sensor_id
108 }
109 }`)
110 req.Header.Set("Cache-Control", "no-cache")
111 for ip := range ipChan {
112 req.Var("ip", ip)
113 if senid != "" {
114 req.Var("senid", senid)
115 }
116 var respData observationResultSet
117 if err := client.Run(context.Background(), req, &respData); err != nil {
118 log.Fatal(err)
119 }
120 if len(respData.Entries) > 0 {
121 for _, entry := range respData.Entries {
122 out, err := json.Marshal(entry)
123 if err != nil {
124 log.Fatal(err)
125 }
126 fmt.Println(string(out))
127 }
128 }
129 }
130 }(&wg)
131 }
132
133 hosts, err := hosts(args[0])
134 if err != nil {
135 log.Fatal(err)
136 }
137 for _, host := range hosts {
138 ipChan <- host
139 }
140 close(ipChan)
141 wg.Wait()
142 },
143 }
144
145 func init() {
146 rootCmd.AddCommand(queryCmd)
147
148 queryCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
149 queryCmd.Flags().StringP("url", "u", "http://localhost:8080/query", "URL of GraphQL interface to query")
150 queryCmd.Flags().StringP("sensor", "s", "", "limit query to observations from single sensor")
151 }
+45
-0
cmd/balboa/cmds/rocksdump.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 "github.com/DCSO/balboa/db"
7 log "github.com/sirupsen/logrus"
8 "github.com/spf13/cobra"
9 )
10
11 var rocksdumpCmd = &cobra.Command{
12 Use: "rocksdump [netmask]",
13 Short: "Dump information from RocksDB",
14 Long: `This command allows directly dump bulk information from RocksDB.`,
15 Run: func(cmd *cobra.Command, args []string) {
16 var err error
17
18 if len(args) == 0 {
19 log.Fatal("needs database path as argument")
20 }
21
22 var verbose bool
23 verbose, err = cmd.Flags().GetBool("verbose")
24 if err != nil {
25 log.Fatal(err)
26 }
27 if verbose {
28 log.SetLevel(log.DebugLevel)
29 }
30
31 rdb, err := db.MakeRocksDBReadonly(args[0])
32 if err != nil {
33 log.Fatal(err)
34 }
35
36 rdb.Dump()
37 },
38 }
39
40 func init() {
41 rootCmd.AddCommand(rocksdumpCmd)
42
43 rocksdumpCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
44 }
+48
-0
cmd/balboa/cmds/root.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 "fmt"
7 "os"
8 "os/user"
9
10 "github.com/spf13/cobra"
11 "github.com/spf13/viper"
12 )
13
14 // rootCmd represents the base command when called without any subcommands
15 var rootCmd = &cobra.Command{
16 Use: "balboa",
17 Short: "BAsic Little Book Of Answers",
18 }
19
20 // Execute runs the root command's Execute method as the Execute
21 // hook for the whole package.
22 func Execute() {
23 if err := rootCmd.Execute(); err != nil {
24 fmt.Println(err)
25 os.Exit(1)
26 }
27 }
28
29 func init() {
30 cobra.OnInitialize(initConfig)
31 }
32
33 func initConfig() {
34 usr, _ := user.Current()
35 homedir := usr.HomeDir
36
37 // Search config in home directory with name ".balboa" (without extension).
38 viper.AddConfigPath(homedir)
39 viper.SetConfigName(".balboa")
40
41 viper.AutomaticEnv() // read in environment variables that match
42
43 // If a config file is found, read it in.
44 if err := viper.ReadInConfig(); err == nil {
45 fmt.Println("Using config file:", viper.ConfigFileUsed())
46 }
47 }
+144
-0
cmd/balboa/cmds/serve.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package cmds
4
5 import (
6 "io/ioutil"
7 "os"
8 "os/signal"
9 "syscall"
10
11 "github.com/DCSO/balboa/db"
12 "github.com/DCSO/balboa/feeder"
13 "github.com/DCSO/balboa/observation"
14 "github.com/DCSO/balboa/query"
15 log "github.com/sirupsen/logrus"
16 "github.com/spf13/cobra"
17 )
18
19 var serveCmd = &cobra.Command{
20 Use: "serve",
21 Short: "Run the balboa server",
22 Long: `This command starts the balboa server, accepting submissions and
23 answering queries.`,
24 Run: func(cmd *cobra.Command, args []string) {
25 var err error
26
27 // handle verbosity
28 var verbose bool
29 verbose, err = cmd.Flags().GetBool("verbose")
30 if err != nil {
31 log.Fatal(err)
32 }
33 if verbose {
34 log.SetLevel(log.DebugLevel)
35 }
36
37 // handle logfile preferences
38 var logfile string
39 logfile, err = cmd.Flags().GetString("logfile")
40 if err != nil {
41 log.Fatal(err)
42 }
43 if logfile != "" {
44 lf, err := os.OpenFile(logfile, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
45 if err != nil {
46 log.Fatal(err)
47 }
48 log.Infof("switching to log file %s", logfile)
49 defer lf.Close()
50 log.SetFormatter(&log.TextFormatter{
51 DisableColors: true,
52 FullTimestamp: true,
53 })
54 log.SetOutput(lf)
55 var logjson bool
56 logjson, err = cmd.Flags().GetBool("logjson")
57 if err != nil {
58 log.Fatal(err)
59 }
60 if logjson {
61 log.SetFormatter(&log.JSONFormatter{})
62 }
63 }
64
65 // Set up database from config file
66 var dbFile string
67 dbFile, err = cmd.Flags().GetString("dbconfig")
68 if err != nil {
69 log.Fatal(err)
70 }
71 cfgYaml, err := ioutil.ReadFile(dbFile)
72 if err != nil {
73 log.Fatal(err)
74 }
75 dbsetup, err := db.LoadSetup(cfgYaml)
76 if err != nil {
77 log.Fatal(err)
78 }
79 db.ObservationDB, err = dbsetup.Run()
80 if err != nil {
81 log.Fatal(err)
82 }
83
84 // Set up feeders from config file
85 var feedersFile string
86 feedersFile, err = cmd.Flags().GetString("feeders")
87 if err != nil {
88 log.Fatal(err)
89 }
90 cfgYaml, err = ioutil.ReadFile(feedersFile)
91 if err != nil {
92 log.Fatal(err)
93 }
94 fsetup, err := feeder.LoadSetup(cfgYaml)
95 if err != nil {
96 log.Fatal(err)
97 }
98 err = fsetup.Run(observation.InChan)
99 if err != nil {
100 log.Fatal(err)
101 }
102
103 // Start processing submissions
104 go db.ObservationDB.ConsumeFeed(observation.InChan)
105
106 // start query server
107 var port int
108 port, err = cmd.Flags().GetInt("port")
109 if err != nil {
110 log.Fatal(err)
111 }
112 gql := query.GraphQLFrontend{}
113 gql.Run(int(port))
114
115 sigChan := make(chan os.Signal, 1)
116 done := make(chan bool, 1)
117 signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
118 go func() {
119 sig := <-sigChan
120 log.Infof("received '%v' signal, shutting down", sig)
121 stopChan := make(chan bool)
122 fsetup.Stop(stopChan)
123 <-stopChan
124 stopChan = make(chan bool)
125 dbsetup.Stop(stopChan)
126 <-stopChan
127 gql.Stop()
128 close(done)
129 }()
130 <-done
131 },
132 }
133
134 func init() {
135 rootCmd.AddCommand(serveCmd)
136
137 serveCmd.Flags().BoolP("verbose", "v", false, "verbose mode")
138 serveCmd.Flags().StringP("dbconfig", "d", "database.yaml", "database configuration file")
139 serveCmd.Flags().StringP("feeders", "f", "feeders.yaml", "feeders configuraion file")
140 serveCmd.Flags().IntP("port", "p", 8080, "port for GraphQL server")
141 serveCmd.Flags().StringP("logfile", "l", "/var/log/balboa.log", "log file path")
142 serveCmd.Flags().BoolP("logjson", "j", true, "output log file as JSON")
143 }
+10
-0
cmd/balboa/main.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package main
4
5 import cmd "github.com/DCSO/balboa/cmd/balboa/cmds"
6
7 func main() {
8 cmd.Execute()
9 }
+9
-0
database.yaml less more
0 database:
1 name: Local RocksDB
2 type: rocksdb
3 db_path: /tmp/balboa
4
5 #database:
6 # name: Local Cassandra
7 # type: cassandra
8 # hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]
+18
-0
db/db.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package db
4
5 import "github.com/DCSO/balboa/observation"
6
7 // ObservationDB is the common DB instance used for this balboa session.
8 var ObservationDB DB
9
10 // DB abstracts a database backend for observation storage.
11 type DB interface {
12 AddObservation(observation.InputObservation) observation.Observation
13 ConsumeFeed(chan observation.InputObservation)
14 Shutdown()
15 TotalCount() (int, error)
16 Search(*string, *string, *string, *string) ([]observation.Observation, error)
17 }
+255
-0
db/db_cassandra.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package db
4
5 import (
6 "time"
7
8 "github.com/DCSO/balboa/observation"
9
10 "github.com/gocql/gocql"
11 log "github.com/sirupsen/logrus"
12 )
13
14 // CassandraDB is a DB implementation based on Apache Cassandra.
15 type CassandraDB struct {
16 Cluster *gocql.ClusterConfig
17 Session *gocql.Session
18 StopChan chan bool
19 Nworkers uint
20 }
21
22 // MakeCassandraDB returns a new CassandraDB instance connecting to the
23 // provided hosts.
24 func MakeCassandraDB(hosts []string, username, password string, nofWorkers uint) (*CassandraDB, error) {
25 cluster := gocql.NewCluster(hosts...)
26 cluster.Keyspace = "balboa"
27 cluster.ProtoVersion = 4
28 cluster.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy())
29 cluster.RetryPolicy = &gocql.ExponentialBackoffRetryPolicy{NumRetries: 5}
30 cluster.Consistency = gocql.Two
31 if len(username) > 0 && len(password) > 0 {
32 cluster.Authenticator = gocql.PasswordAuthenticator{
33 Username: username,
34 Password: password,
35 }
36 }
37 gsession, err := cluster.CreateSession()
38 if err != nil {
39 return nil, err
40 }
41 //session := gockle.NewSession(gsession)
42 db := &CassandraDB{
43 Cluster: cluster,
44 Session: gsession,
45 StopChan: make(chan bool),
46 Nworkers: nofWorkers,
47 }
48 return db, nil
49 }
50
51 //func makeCassandraDBMock() (*CassandraDB, *gockle.SessionMock) {
52 // mock := &gockle.SessionMock{}
53 // db := &CassandraDB{
54 // //Session: mock,
55 // StopChan: make(chan bool),
56 // }
57 // return db, mock
58 //}
59
60 // AddObservation adds a single observation synchronously to the database.
61 func (db *CassandraDB) AddObservation(obs observation.InputObservation) observation.Observation {
62 //Not implemented
63 log.Warn("AddObservation() not yet implemented on Cassandra backend")
64 return observation.Observation{}
65 }
66
67 func (db *CassandraDB) runChunkWorker(inChan chan observation.InputObservation) {
68 rdataUpd := db.Session.Query(`UPDATE observations_by_rdata SET last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;`)
69 rrnameUpd := db.Session.Query(`UPDATE observations_by_rrname SET last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;`)
70 firstseenUpd := db.Session.Query(`INSERT INTO observations_firstseen (first_seen, rrname, rdata, rrtype, sensor_id) values (?, ?, ?, ?, ?) IF NOT EXISTS;`)
71 countsUpd := db.Session.Query(`UPDATE observations_counts SET count = count + ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;`)
72 for obs := range inChan {
73 select {
74 case <-db.StopChan:
75 log.Info("database ingest terminated")
76 return
77 default:
78 if obs.Rdata == "" {
79 obs.Rdata = "-"
80 }
81 if err := rdataUpd.Bind(obs.TimestampEnd, obs.Rdata, obs.Rrname, obs.Rrtype, obs.SensorID).Exec(); err != nil {
82 log.Error(err)
83 continue
84 }
85 if err := rrnameUpd.Bind(obs.TimestampEnd, obs.Rrname, obs.Rdata, obs.Rrtype, obs.SensorID).Exec(); err != nil {
86 log.Error(err)
87 continue
88 }
89 if err := firstseenUpd.Bind(obs.TimestampStart, obs.Rrname, obs.Rdata, obs.Rrtype, obs.SensorID).Exec(); err != nil {
90 log.Error(err)
91 continue
92 }
93 if err := countsUpd.Bind(obs.Count, obs.Rdata, obs.Rrname, obs.Rrtype, obs.SensorID).Exec(); err != nil {
94 log.Error(err)
95 continue
96 }
97 }
98 }
99 }
100
101 // ConsumeFeed accepts observations from a channel and queues them for
102 // database insertion.
103 func (db *CassandraDB) ConsumeFeed(inChan chan observation.InputObservation) {
104 var w uint
105 sendChan := make(chan observation.InputObservation, 500)
106 log.Debugf("Firing up %d workers", db.Nworkers)
107 for w = 1; w <= db.Nworkers; w++ {
108 go db.runChunkWorker(sendChan)
109 }
110 for {
111 select {
112 case <-db.StopChan:
113 log.Info("database ingest terminated")
114 return
115 case obs := <-inChan:
116 sendChan <- obs
117 }
118 }
119 }
120
121 // Search returns a slice of observations matching one or more criteria such
122 // as rdata, rrname, rrtype or sensor ID.
123 func (db *CassandraDB) Search(qrdata, qrrname, qrrtype, qsensorID *string) ([]observation.Observation, error) {
124 outs := make([]observation.Observation, 0)
125 var getQueryString string
126 var rdataFirst, hasSecond bool
127 var getQuery *gocql.Query
128
129 // determine appropriate table and parameterisation for query
130 if qrdata != nil {
131 rdataFirst = true
132 if qrrname != nil {
133 hasSecond = true
134 getQueryString = "SELECT * FROM observations_by_rdata WHERE rdata = ? and rrtype = ?"
135 } else {
136 hasSecond = false
137 getQueryString = "SELECT * FROM observations_by_rdata WHERE rdata = ?"
138 }
139 } else {
140 rdataFirst = false
141 if qrdata != nil {
142 hasSecond = true
143 getQueryString = "SELECT * FROM observations_by_rrname WHERE rrname = ? and rdata = ?"
144 } else {
145 hasSecond = false
146 getQueryString = "SELECT * FROM observations_by_rrname WHERE rrname = ?"
147 }
148 }
149 log.Debug(getQueryString)
150 getQuery = db.Session.Query(getQueryString)
151 getQuery.Consistency(gocql.One)
152
153 // do parameterised search
154 if rdataFirst {
155 if hasSecond {
156 getQuery.Bind(*qrdata, *qrrname)
157 } else {
158 getQuery.Bind(*qrdata)
159 }
160 } else {
161 if hasSecond {
162 getQuery.Bind(*qrrname, *qrdata)
163 } else {
164 getQuery.Bind(*qrrname)
165 }
166 }
167
168 // retrieve hits for initial queries
169 iter := getQuery.Iter()
170 for {
171 row := make(map[string]interface{})
172 if !iter.MapScan(row) {
173 break
174 }
175
176 if rrnameV, ok := row["rrname"]; ok {
177 var rdata, rrname, rrtype, sensorID string
178 var lastSeen int
179 rrname = rrnameV.(string)
180 if rdataV, ok := row["rdata"]; ok {
181 rdata = rdataV.(string)
182
183 // secondary filtering by sensor ID and RRType
184 if sensorIDV, ok := row["sensor_id"]; ok {
185 sensorID = sensorIDV.(string)
186 if qsensorID != nil && *qsensorID != sensorID {
187 continue
188 }
189 }
190 if rrtypeV, ok := row["rrtype"]; ok {
191 rrtype = rrtypeV.(string)
192 if qrrtype != nil && *qrrtype != rrtype {
193 continue
194 }
195 }
196 if lastSeenV, ok := row["last_seen"]; ok {
197 lastSeen = int(lastSeenV.(time.Time).Unix())
198 }
199
200 // we now have a result item
201 out := observation.Observation{
202 RRName: rrname,
203 RData: rdata,
204 RRType: rrtype,
205 SensorID: sensorID,
206 LastSeen: lastSeen,
207 }
208
209 // manual joins -> get additional data from counts table
210 tmpMap := make(map[string]interface{})
211 getCounters := db.Session.Query(`SELECT count FROM observations_counts WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?`).Bind(rrname, rdata, rrtype, sensorID)
212 err := getCounters.MapScan(tmpMap)
213 if err != nil {
214 log.Errorf("getCount: %s", err.Error())
215 continue
216 }
217 out.Count = int(tmpMap["count"].(int64))
218
219 tmpMap = make(map[string]interface{})
220 getFirstSeen := db.Session.Query(`SELECT first_seen FROM observations_firstseen WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?`).Bind(rrname, rdata, rrtype, sensorID)
221 err = getFirstSeen.MapScan(tmpMap)
222 if err != nil {
223 log.Errorf("getFirstSeen: %s", err.Error())
224 continue
225 }
226 out.FirstSeen = int(tmpMap["first_seen"].(int64))
227
228 outs = append(outs, out)
229 } else {
230 log.Warn("result is missing rdata column, something is very wrong")
231 }
232 } else {
233 log.Warn("result is missing rrname column, something is very wrong")
234 }
235
236 }
237
238 return outs, nil
239 }
240
241 // TotalCount returns the overall number of observations across all sensors.
242 func (db *CassandraDB) TotalCount() (int, error) {
243 // TODO
244 return 0, nil
245 }
246
247 // Shutdown closes the database connection, leaving the database unable to
248 // process both reads and writes.
249 func (db *CassandraDB) Shutdown() {
250 close(db.StopChan)
251 if db.Session != nil {
252 db.Session.Close()
253 }
254 }
+134
-0
db/db_cassandra_test.go less more
0 package db
1
2 // func TestCassandraSimpleConsume(t *testing.T) {
3 // t.Skip("skipping Cassandra tests due to removed gockle support")
4 // db, m := makeCassandraDBMock()
5 // defer db.Shutdown()
6
7 // inChan := make(chan observation.InputObservation)
8 // stopChan := make(chan bool)
9
10 // go db.ConsumeFeed(inChan)
11
12 // timeStart := time.Now()
13 // timeEnd := time.Now()
14
15 // cnt := 0
16 // var iteratorMock = &gockle.IteratorMock{}
17 // iteratorMock.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
18 // m["rrname"] = "foo.bar"
19 // m["rrtype"] = "A"
20 // m["rdata"] = "12.34.56.78"
21 // m["count"] = 2
22 // m["sensor_id"] = "deadcafe"
23 // cnt++
24 // return cnt < 4
25 // })
26 // iteratorMock.When("Close").Return(nil)
27
28 // cnt2 := 0
29 // var iteratorMockRdata = &gockle.IteratorMock{}
30 // iteratorMockRdata.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
31 // m["rrname"] = "foo.bar"
32 // m["rrtype"] = "A"
33 // m["rdata"] = "12.34.56.79"
34 // m["count"] = 96
35 // m["sensor_id"] = "deadcafe"
36 // cnt2++
37 // return cnt2 < 2
38 // })
39 // iteratorMockRdata.When("Close").Return(nil)
40
41 // cnt3 := 0
42 // var iteratorMockCnt = &gockle.IteratorMock{}
43 // iteratorMockCnt.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
44 // m["count"] = int64(3)
45 // cnt3++
46 // return cnt3 < 2
47 // })
48 // iteratorMockCnt.When("Close").Return(nil)
49
50 // cnt4 := 0
51 // var iteratorMockFirstItem = &gockle.IteratorMock{}
52 // iteratorMockFirstItem.When("ScanMap", mock.Any).Call(func(m map[string]interface{}) bool {
53 // m["rrname"] = "foo.bar"
54 // cnt4++
55 // return cnt4 < 2
56 // })
57 // iteratorMockFirstItem.When("Close").Return(nil)
58
59 // m.When("ScanIterator", "SELECT * FROM observations_by_rrname WHERE rrname = ?", mock.Any).Return(iteratorMock)
60 // m.When("Exec", "UPDATE observations_by_rdata SET first_seen = ?, last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
61 // m.When("Exec", "UPDATE observations_by_rdata SET last_seen = ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
62 // m.When("Exec", "UPDATE observations_by_rrname SET first_seen = ?, last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
63 // m.When("Exec", "UPDATE observations_by_rrname SET last_seen = ? where rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
64 // m.When("ScanIterator", "SELECT * FROM observations_by_rdata WHERE rdata = ?", []interface{}{"12.34.56.79"}).Return(iteratorMockRdata)
65 // m.When("Exec", "UPDATE observations_counts SET count = count + ? where rdata = ? and rrname = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return()
66 // m.When("ScanIterator", "SELECT rrname FROM observations_by_rrname WHERE rrname = ? and rdata = ? and rrtype = ? and sensor_id = ?;", mock.Any).Return(iteratorMockFirstItem)
67 // m.When("ScanIterator", "SELECT count FROM observations_counts WHERE rrname = ? AND rdata = ? AND rrtype = ? AND sensor_id = ?", mock.Any).Return(iteratorMockCnt)
68 // m.When("Close").Return()
69
70 // o := observation.InputObservation{
71 // Rrname: "foo.bar",
72 // Rrtype: "A",
73 // SensorID: "deadcafe",
74 // Rdata: "12.34.56.78",
75 // TimestampEnd: timeStart,
76 // TimestampStart: timeEnd,
77 // Count: 1,
78 // }
79 // inChan <- o
80
81 // o = observation.InputObservation{
82 // Rrname: "foo.bar",
83 // Rrtype: "MX",
84 // SensorID: "deadcafe",
85 // Rdata: "12.34.56.77",
86 // TimestampEnd: timeStart,
87 // TimestampStart: timeEnd,
88 // Count: 1,
89 // }
90 // inChan <- o
91
92 // for i := 0; i < 500; i++ {
93 // o = observation.InputObservation{
94 // Rrname: "foo.bar",
95 // Rrtype: "NS",
96 // SensorID: "deadcafe",
97 // Rdata: "12.34.56.79",
98 // TimestampEnd: timeStart,
99 // TimestampStart: timeEnd,
100 // Count: 2,
101 // }
102 // inChan <- o
103 // }
104 // close(stopChan)
105
106 // str := "foo.bar"
107 // obs, err := db.Search(nil, &str, nil, nil)
108 // if err != nil {
109 // t.Fatal(err)
110 // }
111 // if len(obs) != 3 {
112 // t.Fatalf("wrong number of results: %d", len(obs))
113 // }
114
115 // str = "12.34.56.79"
116 // obs, err = db.Search(&str, nil, nil, nil)
117 // if err != nil {
118 // t.Fatal(err)
119 // }
120 // if len(obs) != 1 {
121 // t.Fatalf("wrong number of results: %d", len(obs))
122 // }
123
124 // str = "12.34.56.79"
125 // obs, err = db.Search(&str, nil, nil, &str)
126 // if err != nil {
127 // t.Fatal(err)
128 // }
129 // if len(obs) != 0 {
130 // t.Fatalf("wrong number of results: %d", len(obs))
131 // }
132
133 // }
+94
-0
db/db_config.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package db
4
5 import (
6 "fmt"
7 "runtime"
8
9 log "github.com/sirupsen/logrus"
10 yaml "gopkg.in/yaml.v2"
11 )
12
13 // Setup represents a database backend configuration, typically provided via
14 // YAML.
15 type Setup struct {
16 Database struct {
17 Name string `yaml:"name"`
18 Type string `yaml:"type"`
19 // for local storage
20 DBPath string `yaml:"db_path"`
21 // for RocksDB
22 MemtableMemBudget uint64 `yaml:"membudget"`
23 // for Cassandra
24 Hosts []string `yaml:"hosts"`
25 Username string `yaml:"username"`
26 Password string `yaml:"password"`
27 Nworkers uint `yaml:"nof_workers"`
28 } `yaml:"database"`
29 LoadedDB DB
30 }
31
32 // LoadSetup parses a given YAML description into a new Setup structure.
33 func LoadSetup(in []byte) (*Setup, error) {
34 var s Setup
35 err := yaml.Unmarshal(in, &s)
36 if err != nil {
37 return nil, err
38 }
39 if s.Database.Name == "" {
40 return nil, fmt.Errorf("database name missing")
41 }
42 if s.Database.Type == "" {
43 return nil, fmt.Errorf("database type missing")
44 }
45 switch s.Database.Type {
46 case "rocksdb":
47 if len(s.Database.DBPath) == 0 {
48 return nil, fmt.Errorf("%s: local database path missing", s.Database.Name)
49 }
50 if s.Database.MemtableMemBudget == 0 {
51 log.Infof("%s: memory budget empty, using default of 128MB", s.Database.Name)
52 s.Database.MemtableMemBudget = 128 * 1024 * 1024
53 }
54 case "cassandra":
55 if len(s.Database.Hosts) == 0 {
56 return nil, fmt.Errorf("%s: no Cassandra hosts defined", s.Database.Name)
57 }
58 if s.Database.Nworkers == 0 {
59 s.Database.Nworkers = (uint)(runtime.NumCPU() * 8)
60 log.Infof("%s: number of workers is 0 or undefined, will use default of %d", s.Database.Name, s.Database.Nworkers)
61 }
62 }
63 return &s, nil
64 }
65
66 // Run creates the necessary database objects and makes them ready to
67 // consume or provide data.
68 func (s *Setup) Run() (DB, error) {
69 log.Infof("starting database %s", s.Database.Name)
70 var db DB
71 var err error
72 switch s.Database.Type {
73 case "rocksdb":
74 db, err = MakeRocksDB(s.Database.DBPath, s.Database.MemtableMemBudget)
75 if err != nil {
76 return nil, err
77 }
78 case "cassandra":
79 db, err = MakeCassandraDB(s.Database.Hosts, s.Database.Username, s.Database.Password, s.Database.Nworkers)
80 if err != nil {
81 return nil, err
82 }
83 }
84 s.LoadedDB = db
85 return db, nil
86 }
87
88 // Stop causes a given Setup to shut down the corresponding database
89 // connections defined within.
90 func (s *Setup) Stop(stopChan chan bool) {
91 s.LoadedDB.Shutdown()
92 close(stopChan)
93 }
+105
-0
db/db_config_test.go less more
0 package db
1
2 import (
3 "fmt"
4 "io/ioutil"
5 "os"
6 "testing"
7 )
8
9 func TestDBConfigLoad(t *testing.T) {
10 _, err := LoadSetup([]byte("foo["))
11 if err == nil {
12 t.Fatal("did not error with invalid data")
13 }
14
15 _, err = LoadSetup([]byte(""))
16 if err == nil {
17 t.Fatal("did not error with empty config")
18 }
19
20 _, err = LoadSetup([]byte(`
21 database:
22 name: Whatever
23 `))
24 if err == nil {
25 t.Fatal("did not error with missing db type")
26 }
27
28 _, err = LoadSetup([]byte(`
29 database:
30 name: Local RocksDB
31 type: rocksdb
32 `))
33 if err == nil {
34 t.Fatal("did not error with missing path")
35 }
36
37 _, err = LoadSetup([]byte(`
38 database:
39 name: Local RocksDB
40 type: rocksdb
41 db_path: /tmp/balboa
42 `))
43 if err != nil {
44 t.Fatal("does not accept regular config: ", err)
45 }
46
47 _, err = LoadSetup([]byte(`
48 database:
49 name: Local RocksDB
50 type: rocksdb
51 db_path: /tmp/balboa
52 membudget: 1000000
53 `))
54 if err != nil {
55 t.Fatal("does not accept regular config: ", err)
56 }
57
58 _, err = LoadSetup([]byte(`
59 database:
60 name: Local Cassandra
61 type: cassandra
62 hosts: [ "127.0.0.1", "127.0.0.2", "127.0.0.3" ]
63 `))
64 if err != nil {
65 t.Fatal("does not accept regular config: ", err)
66 }
67
68 _, err = LoadSetup([]byte(`
69 database:
70 name: Local Cassandra
71 type: cassandra
72 `))
73 if err == nil {
74 t.Fatal("did not error with missing hosts")
75 }
76 }
77
78 func TestDBConfigRunRocks(t *testing.T) {
79 dbdir, err := ioutil.TempDir("", "example")
80 if err != nil {
81 t.Fatal(err)
82 }
83 dbs, err := LoadSetup([]byte(fmt.Sprintf(`
84 database:
85 name: Local RocksDB
86 type: rocksdb
87 membudget: 1000000
88 db_path: %s
89 `, dbdir)))
90 if err != nil {
91 t.Fatal("does not accept regular config: ", err)
92 }
93
94 _, err = dbs.Run()
95 if err != nil {
96 t.Fatal(err)
97 }
98 defer os.RemoveAll(dbdir)
99
100 stopChan := make(chan bool)
101 dbs.Stop(stopChan)
102 <-stopChan
103
104 }
+189
-0
db/db_mysql.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package db
4
5 import (
6 "database/sql"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10
11 // imported for side effects
12 _ "github.com/go-sql-driver/mysql"
13 log "github.com/sirupsen/logrus"
14 )
15
16 // create table observations (
17 // id bigint AUTO_INCREMENT,
18 // sensorID varchar(255) NOT NULL,
19 // rrname varchar(255) NOT NULL,
20 // rdata varchar(255) NOT NULL,
21 // rrcode varchar(64) NULL,
22 // rrtype varchar(64) NULL,
23 // count int NOT NULL,
24 // time_first timestamp NOT NULL,
25 // time_last timestamp NOT NULL,
26 // PRIMARY KEY (id),
27 // UNIQUE KEY (rdata, rrname, sensorID),
28 // INDEX rrname (rrname)
29 // );
30
31 // MySQLDB is a DB implementation based on MySQL.
32 type MySQLDB struct {
33 DB *sql.DB
34 }
35
36 const (
37 mysqlbufsize = 50
38 upsertSQL = `
39 INSERT INTO observations
40 (sensorID, rrname, rdata, rrcode, rrtype, count, time_first, time_last)
41 VALUES
42 (?, ?, ?, ?, ?, ?, ?, ?)
43 ON DUPLICATE KEY UPDATE
44 count = count + VALUES(count),
45 time_last = VALUES(time_last);`
46 qrySQL = `
47 SELECT
48 sensorID, rrname, rdata, rrtype, count, time_first, time_last
49 FROM
50 observations
51 WHERE
52 rdata like ? AND rrname like ? AND rrtype like ? AND sensorID like ?;`
53 )
54
55 // MakeMySQLDB returns a new MySQLDB instance based on the given sql.DB
56 // pointer.
57 func MakeMySQLDB(pdb *sql.DB) *MySQLDB {
58 db := &MySQLDB{
59 DB: pdb,
60 }
61 return db
62 }
63
64 // AddObservation adds a single observation synchronously to the database.
65 func (db *MySQLDB) AddObservation(obs observation.InputObservation) observation.Observation {
66 //Not implemented
67 log.Warn("AddObservation() not implemented on MySQL DB backend")
68 return observation.Observation{}
69 }
70
71 // ConsumeFeed accepts observations from a channel and queues them for
72 // database insertion.
73 func (db *MySQLDB) ConsumeFeed(inChan chan observation.InputObservation, stopChan chan bool) {
74 buf := make([]observation.InputObservation, mysqlbufsize)
75 i := 0
76 for {
77 select {
78 case <-stopChan:
79 log.Info("database ingest terminated")
80 return
81 default:
82 if i < mysqlbufsize {
83 log.Debug("buffering ", i)
84 buf[i] = <-inChan
85 i++
86 } else {
87 // flushing buffer in one transaction
88 i = 0
89 startTime := time.Now()
90 tx, err := db.DB.Begin()
91 if err != nil {
92 log.Warn(err)
93 }
94 var upsert *sql.Stmt
95 upsert, _ = tx.Prepare(upsertSQL)
96 for _, obs := range buf {
97 _, err := upsert.Exec(obs.SensorID, obs.Rrname,
98 obs.Rdata, nil, obs.Rrtype, obs.Count,
99 obs.TimestampStart, obs.TimestampEnd)
100 if err != nil {
101 log.Warn(err)
102 tx.Rollback()
103 }
104 }
105 tx.Commit()
106 log.Infof("insert Tx took %v", time.Since(startTime))
107 }
108 }
109 }
110 }
111
112 // Search returns a slice of observations matching one or more criteria such
113 // as rdata, rrname, rrtype or sensor ID.
114 func (db *MySQLDB) Search(qrdata, qrrname, qrrtype, qsensorID *string) ([]observation.Observation, error) {
115 outs := make([]observation.Observation, 0)
116 stmt, err := db.DB.Prepare(qrySQL)
117 if err != nil {
118 return nil, err
119 }
120
121 rdata := "%"
122 if qrdata != nil {
123 rdata = *qrdata
124 }
125 sensorID := "%"
126 if qsensorID != nil {
127 sensorID = *qsensorID
128 }
129 rrname := "%"
130 if qrrname != nil {
131 rrname = *qrrname
132 }
133 rrtype := "%"
134 if qrrtype != nil {
135 rrtype = *qrrtype
136 }
137
138 rows, err := stmt.Query(rdata, rrname, rrtype, sensorID)
139 if err != nil {
140 return nil, err
141 }
142
143 for rows.Next() {
144 out := observation.Observation{}
145 var lastseen, firstseen time.Time
146 err := rows.Scan(&out.SensorID, &out.RRName, &out.RData, &out.RRType,
147 &out.Count, &firstseen, &lastseen)
148 if err != nil {
149 return nil, err
150 }
151 out.FirstSeen = int(firstseen.Unix())
152 out.LastSeen = int(lastseen.Unix())
153 outs = append(outs, out)
154 }
155 err = rows.Err()
156 if err != nil {
157 return nil, err
158 }
159 return outs, nil
160 }
161
162 // TotalCount returns the overall number of observations across all sensors.
163 func (db *MySQLDB) TotalCount() (int, error) {
164 var val int
165 var err error
166 rows, err := db.DB.Query("SELECT count(*) FROM observations")
167 if err != nil {
168 return 0, err
169 }
170 defer rows.Close()
171 for rows.Next() {
172 err := rows.Scan(&val)
173 if err != nil {
174 return 0, err
175 }
176 }
177 err = rows.Err()
178 if err != nil {
179 return 0, err
180 }
181 return val, nil
182 }
183
184 // Shutdown closes the database connection, leaving the database unable to
185 // process both reads and writes.
186 func (db *MySQLDB) Shutdown() {
187 db.DB.Close()
188 }
+152
-0
db/db_mysql_test.go less more
0 package db
1
2 import (
3 "testing"
4 "time"
5
6 sqlmock "github.com/DATA-DOG/go-sqlmock"
7 "github.com/DCSO/balboa/observation"
8 )
9
10 func createMysqlTmpDB(t *testing.T) (*MySQLDB, sqlmock.Sqlmock) {
11 db, mock, err := sqlmock.New()
12 if err != nil {
13 t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
14 }
15
16 return MakeMySQLDB(db), mock
17 }
18
19 func TestMySQLSimpleConsume(t *testing.T) {
20 db, mock := createMysqlTmpDB(t)
21 defer db.Shutdown()
22
23 inChan := make(chan observation.InputObservation)
24 stopChan := make(chan bool)
25
26 go db.ConsumeFeed(inChan, stopChan)
27
28 timeStart := time.Now()
29 timeEnd := time.Now()
30 mock.ExpectBegin()
31 mock.ExpectPrepare("INSERT INTO observations")
32 mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "x.foo.bar", "12.34.56.78", nil, "A", 1, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
33 mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "y.foo.bar", "12.34.56.77", nil, "MX", 1, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
34 for i := 0; i < 48; i++ {
35 mock.ExpectExec("INSERT INTO observations").WithArgs("deadcafe", "foo.bar", "12.34.56.79", nil, "NS", 2, timeStart, timeEnd).WillReturnResult(sqlmock.NewResult(1, 1))
36 }
37 mock.ExpectPrepare("SELECT (.+) FROM observations")
38 retRows := sqlmock.NewRows(
39 []string{
40 "sensor_id",
41 "rrname",
42 "rdata",
43 "rrtype",
44 "count",
45 "time_first",
46 "time_last",
47 },
48 )
49 retRows.AddRow("12.34.56.78", "A", "x.foo.bar", "deadcafe", 1, timeStart, timeEnd)
50 retRows.AddRow("12.34.56.78", "NX", "y.foo.bar", "deadcafe", 1, timeStart, timeEnd)
51 retRows.AddRow("12.34.56.78", "NS", "foo.bar", "deadcafe", 96, timeStart, timeEnd)
52 mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
53 mock.ExpectPrepare("SELECT (.+) FROM observations")
54 retRows = sqlmock.NewRows(
55 []string{
56 "sensor_id",
57 "rrname",
58 "rdata",
59 "rrtype",
60 "count",
61 "time_first",
62 "time_last",
63 },
64 )
65 retRows.AddRow("12.34.56.78", "NS", "foo.bar", "deadcafe", 96, timeStart, timeEnd)
66 mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
67 mock.ExpectPrepare("SELECT (.+) FROM observations")
68 retRows = sqlmock.NewRows(
69 []string{
70 "sensor_id",
71 "rrname",
72 "rdata",
73 "rrtype",
74 "count",
75 "time_first",
76 "time_last",
77 },
78 )
79 mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(retRows)
80 mock.ExpectQuery("SELECT (.+) FROM observations").WillReturnRows(sqlmock.NewRows(
81 []string{
82 "count",
83 }).AddRow(3))
84
85 inChan <- observation.InputObservation{
86 Rrname: "x.foo.bar",
87 Rrtype: "A",
88 SensorID: "deadcafe",
89 Rdata: "12.34.56.78",
90 TimestampEnd: timeEnd,
91 TimestampStart: timeStart,
92 Count: 1,
93 }
94 inChan <- observation.InputObservation{
95 Rrname: "y.foo.bar",
96 Rrtype: "MX",
97 SensorID: "deadcafe",
98 Rdata: "12.34.56.77",
99 TimestampEnd: timeEnd,
100 TimestampStart: timeStart,
101 Count: 1,
102 }
103 for i := 0; i < 50; i++ {
104 inChan <- observation.InputObservation{
105 Rrname: "foo.bar",
106 Rrtype: "NS",
107 SensorID: "deadcafe",
108 Rdata: "12.34.56.79",
109 TimestampEnd: timeEnd,
110 TimestampStart: timeStart,
111 Count: 2,
112 }
113 }
114 close(stopChan)
115
116 str := "foo.bar"
117 obs, err := db.Search(nil, &str, nil, nil)
118 if err != nil {
119 t.Fatal(err)
120 }
121 if len(obs) != 3 {
122 t.Fatalf("wrong number of results: %d", len(obs))
123 }
124
125 str = "12.34.56.79"
126 obs, err = db.Search(&str, nil, nil, nil)
127 if err != nil {
128 t.Fatal(err)
129 }
130 if len(obs) != 1 {
131 t.Fatalf("wrong number of results: %d", len(obs))
132 }
133
134 str = "12.34.56.79"
135 obs, err = db.Search(&str, nil, nil, &str)
136 if err != nil {
137 t.Fatal(err)
138 }
139 if len(obs) != 0 {
140 t.Fatalf("wrong number of results: %d", len(obs))
141 }
142
143 tc, err := db.TotalCount()
144 if err != nil {
145 t.Fatal(err)
146 }
147 if tc != 3 {
148 t.Fatalf("wrong number of results: %d", len(obs))
149 }
150
151 }
+312
-0
db/db_rocksdb.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package db
4
5 // #cgo LDFLAGS: -lrocksdb -ltpl
6 // #cgo CFLAGS: -O3 -Wno-implicit-function-declaration -Wall -Wextra -Wno-unused-parameter
7 // #include "obs_rocksdb.h"
8 // #include <stdlib.h>
9 import "C"
10
11 import (
12 "encoding/json"
13 "fmt"
14 "strings"
15 "time"
16 "unsafe"
17
18 "github.com/DCSO/balboa/observation"
19
20 log "github.com/sirupsen/logrus"
21 )
22
23 const (
24 rocksTxSize = 50
25 keySepChar = "\x1f" // we use ASCII Unit separators
26 )
27
28 // RocksDB is a DB implementation based on Facebook's RocksDB engine.
29 type RocksDB struct {
30 db *C.ObsDB
31 stopChan chan bool
32 }
33
34 func rocksMakeKey(sensor string, rrname string, rrtype string, rdata string) string {
35 k := fmt.Sprintf("o%s%s%s%s%s%s%s%s", keySepChar, rrname, keySepChar, sensor, keySepChar, rrtype, keySepChar, rdata)
36 return k
37 }
38
39 func rocksMakeInvKey(sensor string, rrname string, rrtype string, rdata string) string {
40 k := fmt.Sprintf("i%s%s%s%s%s%s%s%s", keySepChar, rdata, keySepChar, sensor, keySepChar, rrname, keySepChar, rrtype)
41 return k
42 }
43
44 // MakeRocksDB returns a new RocksDB instance, storing data at the given dbPath.
45 // The second parameter specifies a memory budget which determines the size of
46 // memtables and caches.
47 func MakeRocksDB(dbPath string, membudget uint64) (*RocksDB, error) {
48 log.Info("opening database...")
49
50 e := C.error_new()
51 defer C.error_delete(e)
52 cdbPath := C.CString(dbPath)
53 cdb := C.obsdb_open(cdbPath, C.size_t(membudget), e)
54 defer C.free(unsafe.Pointer(cdbPath))
55
56 if C.error_is_set(e) {
57 return nil, fmt.Errorf("%s", C.GoString(C.error_get(e)))
58 }
59
60 db := &RocksDB{
61 db: cdb,
62 stopChan: make(chan bool),
63 }
64
65 log.Info("database ready")
66 return db, nil
67 }
68
69 // MakeRocksDBReadonly returns a new read-only RocksDB instance.
70 func MakeRocksDBReadonly(dbPath string) (*RocksDB, error) {
71 log.Info("opening database...")
72
73 e := C.error_new()
74 defer C.error_delete(e)
75 cdbPath := C.CString(dbPath)
76 cdb := C.obsdb_open_readonly(cdbPath, e)
77 defer C.free(unsafe.Pointer(cdbPath))
78
79 if C.error_is_set(e) {
80 return nil, fmt.Errorf("%s", C.GoString(C.error_get(e)))
81 }
82
83 db := &RocksDB{
84 db: cdb,
85 stopChan: make(chan bool),
86 }
87
88 log.Info("database ready")
89 return db, nil
90 }
91
92 func rocksTxDedup(in []observation.InputObservation) []observation.InputObservation {
93 cache := make(map[string]*observation.InputObservation)
94 for i, inObs := range in {
95 key := rocksMakeKey(inObs.SensorID, inObs.Rrname, inObs.Rrtype, inObs.Rdata)
96 _, ok := cache[key]
97 if ok {
98 cache[key].Count += inObs.Count
99 cache[key].TimestampEnd = inObs.TimestampEnd
100 } else {
101 cache[key] = &in[i]
102 }
103 }
104 out := make([]observation.InputObservation, 0)
105 for _, v := range cache {
106 out = append(out, *v)
107 }
108 if len(in) != len(out) {
109 log.Debugf("TX dedup: %d -> %d", len(in), len(out))
110 }
111 return out
112 }
113
114 // AddObservation adds a single observation synchronously to the database.
115 func (db *RocksDB) AddObservation(obs observation.InputObservation) observation.Observation {
116 var cobs C.Observation
117 e := C.error_new()
118 defer C.error_delete(e)
119
120 key := rocksMakeKey(obs.SensorID, obs.Rrname, obs.Rrtype,
121 obs.Rdata)
122 invKey := rocksMakeInvKey(obs.SensorID, obs.Rrname,
123 obs.Rrtype, obs.Rdata)
124
125 cobs.key = C.CString(key)
126 cobs.inv_key = C.CString(invKey)
127 cobs.count = C.uint(obs.Count)
128 cobs.last_seen = C.uint(obs.TimestampEnd.Unix())
129 cobs.first_seen = C.uint(obs.TimestampStart.Unix())
130
131 C.obsdb_put(db.db, &cobs, e)
132 if C.error_is_set(e) {
133 log.Errorf("%s", C.GoString(C.error_get(e)))
134 }
135
136 C.free(unsafe.Pointer(cobs.key))
137 C.free(unsafe.Pointer(cobs.inv_key))
138
139 r, err := db.Search(&obs.Rdata, &obs.Rrname, &obs.Rrtype, &obs.SensorID)
140 if err != nil {
141 log.Error(err)
142 }
143 return r[0]
144 }
145
146 type rocksObservation struct {
147 Count int
148 LastSeen int64
149 FirstSeen int64
150 }
151
152 // ConsumeFeed accepts observations from a channel and queues them for
153 // database insertion.
154 func (db *RocksDB) ConsumeFeed(inChan chan observation.InputObservation) {
155 var i = 0
156 buf := make([]observation.InputObservation, rocksTxSize)
157 var cobs C.Observation
158 e := C.error_new()
159 defer C.error_delete(e)
160
161 for {
162 select {
163 case <-db.stopChan:
164 log.Info("database ingest terminated")
165 return
166 default:
167 obs := <-inChan
168 if i < rocksTxSize {
169 buf[i] = obs
170 i++
171 } else {
172 i = 0
173 startTime := time.Now()
174 for _, obs := range rocksTxDedup(buf) {
175 if obs.Rrtype == "" {
176 continue
177 }
178
179 key := rocksMakeKey(obs.SensorID, obs.Rrname, obs.Rrtype,
180 obs.Rdata)
181 invKey := rocksMakeInvKey(obs.SensorID, obs.Rrname,
182 obs.Rrtype, obs.Rdata)
183
184 cobs.key = C.CString(key)
185 cobs.inv_key = C.CString(invKey)
186 cobs.count = C.uint(obs.Count)
187 cobs.last_seen = C.uint(obs.TimestampEnd.Unix())
188 cobs.first_seen = C.uint(obs.TimestampStart.Unix())
189
190 C.obsdb_put(db.db, &cobs, e)
191 if C.error_is_set(e) {
192 log.Errorf("%s", C.GoString(C.error_get(e)))
193 }
194
195 C.free(unsafe.Pointer(cobs.key))
196 C.free(unsafe.Pointer(cobs.inv_key))
197 }
198 log.Debugf("insert Tx took %v", time.Since(startTime))
199 }
200 }
201 }
202 }
203
204 //export cgoLogInfo
205 func cgoLogInfo(str *C.char) {
206 log.Info(C.GoString(str))
207 }
208
209 //export cgoLogDebug
210 func cgoLogDebug(str *C.char) {
211 log.Debug(C.GoString(str))
212 }
213
214 // Search returns a slice of observations matching one or more criteria such
215 // as rdata, rrname, rrtype or sensor ID.
216 func (db *RocksDB) Search(qrdata, qrrname, qrrtype, qsensorID *string) ([]observation.Observation, error) {
217 outs := make([]observation.Observation, 0)
218 var cqrdata, cqrrname, cqrrtype, cqsensorID *C.char
219 var i uint
220
221 if qrdata == nil {
222 cqrdata = nil
223 } else {
224 cqrdata = C.CString(*qrdata)
225 defer C.free(unsafe.Pointer(cqrdata))
226 }
227 if qrrname == nil {
228 cqrrname = nil
229 } else {
230 cqrrname = C.CString(*qrrname)
231 defer C.free(unsafe.Pointer(cqrrname))
232 }
233 if qrrtype == nil {
234 cqrrtype = nil
235 } else {
236 cqrrtype = C.CString(*qrrtype)
237 defer C.free(unsafe.Pointer(cqrrtype))
238 }
239 if qsensorID == nil {
240 cqsensorID = nil
241 } else {
242 cqsensorID = C.CString(*qsensorID)
243 defer C.free(unsafe.Pointer(cqsensorID))
244 }
245
246 r := C.obsdb_search(db.db, cqrdata, cqrrname, cqrrtype, cqsensorID)
247 defer C.obs_set_delete(r)
248
249 for i = 0; i < uint(C.obs_set_size(r)); i++ {
250 o := C.obs_set_get(r, C.ulong(i))
251 valArr := strings.Split(C.GoString(o.key), keySepChar)
252 outs = append(outs, observation.Observation{
253 SensorID: valArr[2],
254 Count: int(o.count),
255 RData: valArr[4],
256 RRName: valArr[1],
257 RRType: valArr[3],
258 FirstSeen: int(o.first_seen),
259 LastSeen: int(o.last_seen),
260 })
261 }
262
263 return outs, nil
264 }
265
266 //export cgoObsDump
267 func cgoObsDump(o *C.Observation) {
268 valArr := strings.Split(C.GoString(o.key), keySepChar)
269 myObs := observation.Observation{
270 SensorID: valArr[2],
271 Count: int(o.count),
272 RData: valArr[4],
273 RRName: valArr[1],
274 RRType: valArr[3],
275 FirstSeen: int(o.first_seen),
276 LastSeen: int(o.last_seen),
277 }
278 js, err := json.Marshal(myObs)
279 if err == nil {
280 fmt.Println(string(js))
281 }
282 }
283
284 // Dump prints all aggregated observations in the database to stdout, in JSON format.
285 func (db *RocksDB) Dump() error {
286 e := C.error_new()
287 defer C.error_delete(e)
288
289 C.obsdb_dump(db.db, e)
290
291 if C.error_is_set(e) {
292 return fmt.Errorf("%s", C.GoString(C.error_get(e)))
293 }
294
295 return nil
296 }
297
298 // TotalCount returns the overall number of observations across all sensors.
299 func (db *RocksDB) TotalCount() (int, error) {
300 var val int
301 val = int(C.obsdb_num_keys(db.db))
302 return val, nil
303 }
304
305 // Shutdown closes the database connection, leaving the database unable to
306 // process both reads and writes.
307 func (db *RocksDB) Shutdown() {
308 close(db.stopChan)
309 C.obsdb_close(db.db)
310 log.Info("database closed")
311 }
+190
-0
db/db_rocksdb_test.go less more
0 package db
1
2 import (
3 "io/ioutil"
4 "os"
5 "syscall"
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 )
11
12 func createRocksTmpDB(t *testing.T) (*RocksDB, string) {
13 dbdir, err := ioutil.TempDir("", "example")
14 if err != nil {
15 t.Fatal(err)
16 }
17 db, err := MakeRocksDB(dbdir, 8000000)
18 if err != nil {
19 t.Fatal(err)
20 }
21 if db == nil {
22 t.Fatal("no db created")
23 }
24 return db, dbdir
25 }
26
27 func TestRocksCreate(t *testing.T) {
28 db, dbdir := createRocksTmpDB(t)
29 defer os.RemoveAll(dbdir)
30 defer db.Shutdown()
31 }
32
33 func TestRocksCreateFail(t *testing.T) {
34 // skip this test if run as root
35 if syscall.Getuid() == 0 {
36 t.Skip()
37 }
38 _, err := MakeRocksDB("/nonexistent", 8000000)
39 if err == nil {
40 t.Fatal(err)
41 }
42 }
43
44 func TestRocksSimpleStore(t *testing.T) {
45 db, dbdir := createRocksTmpDB(t)
46 defer os.RemoveAll(dbdir)
47 defer db.Shutdown()
48
49 a := db.AddObservation(observation.InputObservation{
50 Rrname: "foo.bar",
51 Rrtype: "A",
52 SensorID: "deadcafe",
53 Rdata: "12.34.56.78",
54 TimestampEnd: time.Now(),
55 TimestampStart: time.Now(),
56 Count: 1,
57 })
58 if a.RData != "12.34.56.78" || a.RRName != "foo.bar" || a.RRType != "A" ||
59 a.Count != 1 {
60 t.Fatal("invalid return")
61 }
62 a = db.AddObservation(observation.InputObservation{
63 Rrname: "foo.bar",
64 Rrtype: "MX",
65 SensorID: "deadcafe",
66 Rdata: "12.34.56.77",
67 TimestampEnd: time.Now(),
68 TimestampStart: time.Now(),
69 Count: 1,
70 })
71 if a.RData != "12.34.56.77" || a.RRName != "foo.bar" || a.RRType != "MX" ||
72 a.Count != 1 {
73 t.Fatal("invalid return")
74 }
75 a = db.AddObservation(observation.InputObservation{
76 Rrname: "foo.bar",
77 Rrtype: "NS",
78 SensorID: "deadcafe",
79 Rdata: "12.34.56.79",
80 TimestampEnd: time.Now(),
81 TimestampStart: time.Now(),
82 Count: 1,
83 })
84 if a.RData != "12.34.56.79" || a.RRName != "foo.bar" || a.RRType != "NS" ||
85 a.Count != 1 {
86 t.Fatal("invalid return")
87 }
88
89 str := "foo.bar"
90 obs, err := db.Search(nil, &str, nil, nil)
91 if err != nil {
92 t.Fatal(err)
93 }
94 if len(obs) != 3 {
95 t.Fatalf("wrong number of results: %d", len(obs))
96 }
97
98 a = db.AddObservation(observation.InputObservation{
99 Rrname: "foo.bar",
100 Rrtype: "NS",
101 SensorID: "deadcafe",
102 Rdata: "12.34.56.79",
103 TimestampEnd: time.Now().Add(77 * time.Second),
104 TimestampStart: time.Now().Add(77 * time.Second),
105 Count: 4,
106 })
107 if a.RData != "12.34.56.79" || a.RRName != "foo.bar" || a.RRType != "NS" ||
108 a.Count != 5 {
109 t.Fatalf("invalid return %v", a)
110 }
111
112 obs, err = db.Search(nil, &str, nil, nil)
113 if err != nil {
114 t.Fatal(err)
115 }
116 if len(obs) != 3 {
117 t.Fatalf("wrong number of results: %d", len(obs))
118 }
119 }
120
121 func TestRocksSimpleConsume(t *testing.T) {
122 db, dbdir := createRocksTmpDB(t)
123 defer os.RemoveAll(dbdir)
124 defer db.Shutdown()
125
126 inChan := make(chan observation.InputObservation)
127 stopChan := make(chan bool)
128
129 go db.ConsumeFeed(inChan)
130
131 inChan <- observation.InputObservation{
132 Rrname: "foo.bar",
133 Rrtype: "A",
134 SensorID: "deadcafe",
135 Rdata: "12.34.56.78",
136 TimestampEnd: time.Now(),
137 TimestampStart: time.Now(),
138 Count: 1,
139 }
140 inChan <- observation.InputObservation{
141 Rrname: "foo.bar",
142 Rrtype: "MX",
143 SensorID: "deadcafe",
144 Rdata: "12.34.56.77",
145 TimestampEnd: time.Now(),
146 TimestampStart: time.Now(),
147 Count: 1,
148 }
149 for i := 0; i < 10000; i++ {
150 inChan <- observation.InputObservation{
151 Rrname: "foo.bar",
152 Rrtype: "NS",
153 SensorID: "deadcafe",
154 Rdata: "12.34.56.79",
155 TimestampEnd: time.Now(),
156 TimestampStart: time.Now(),
157 Count: 2,
158 }
159 }
160 close(stopChan)
161
162 str := "foo.bar"
163 obs, err := db.Search(nil, &str, nil, nil)
164 if err != nil {
165 t.Fatal(err)
166 }
167 if len(obs) != 3 {
168 t.Fatalf("wrong number of results: %d", len(obs))
169 }
170
171 str = "12.34.56.79"
172 obs, err = db.Search(&str, nil, nil, nil)
173 if err != nil {
174 t.Fatal(err)
175 }
176 if len(obs) != 1 {
177 t.Fatalf("wrong number of results: %d", len(obs))
178 }
179
180 str = "12.34.56.79"
181 obs, err = db.Search(&str, nil, nil, &str)
182 if err != nil {
183 t.Fatal(err)
184 }
185 if len(obs) != 0 {
186 t.Fatalf("wrong number of results: %d", len(obs))
187 }
188
189 }
+699
-0
db/obs_rocksdb.c less more
0 /*
1 balboa
2 Copyright (c) 2018, DCSO GmbH
3 */
4
5 #include "obs_rocksdb.h"
6
7 #include <stdlib.h>
8 #include <stdio.h>
9 #include <string.h>
10 #include "rocksdb/c.h"
11 #include "tpl.h"
12
13 extern void cgoLogInfo(const char*);
14 extern void cgoLogDebug(const char*);
15 extern void cgoObsDump(Observation*);
16
17 #define OBS_SET_STEP_SIZE 1000
18
19 struct Error {
20 char *msg;
21 };
22
23 Error* error_new()
24 {
25 Error *e = calloc(1, sizeof(Error));
26 if (!e) {
27 return NULL;
28 }
29 return e;
30 }
31
32 const char* error_get(Error *e)
33 {
34 if (!e)
35 return NULL;
36 return e->msg;
37 }
38
39 static void error_set(Error *e, const char *msg)
40 {
41 if (!e)
42 return;
43 if (!msg)
44 return;
45 if (e->msg)
46 free(e->msg);
47 e->msg = strdup(msg);
48 }
49
50 void error_unset(Error *e)
51 {
52 if (!e)
53 return;
54 if (e->msg)
55 free(e->msg);
56 e->msg = NULL;
57 }
58
59 bool error_is_set(Error *e)
60 {
61 if (!e)
62 return NULL;
63 return (e->msg != NULL);
64 }
65
66 void error_delete(Error *e)
67 {
68 if (!e)
69 return;
70 if (e->msg)
71 free(e->msg);
72 free(e);
73 }
74
75
76 struct ObsSet {
77 unsigned long size, used;
78 Observation **os;
79 };
80
81 static ObsSet* obs_set_create(unsigned long size)
82 {
83 ObsSet *os = calloc((size_t) 1, sizeof(ObsSet));
84 if (!os)
85 return NULL;
86 os->size = size;
87 os->used = 0;
88 os->os = calloc((size_t) size, sizeof(Observation*));
89 if (os->os == NULL) {
90 free(os);
91 return NULL;
92 }
93 return os;
94 }
95
96 unsigned long obs_set_size(ObsSet *os)
97 {
98 if (!os)
99 return 0;
100 return os->used;
101 }
102
103 static void obs_set_add(ObsSet *os, Observation *o)
104 {
105 if (!os || !os->os)
106 return;
107 if (os->used == os->size) {
108 os->size += OBS_SET_STEP_SIZE;
109 os->os = realloc(os->os, (size_t) (os->size * sizeof(Observation*)));
110 }
111 os->os[os->used++] = o;
112 }
113
114 const Observation* obs_set_get(ObsSet *os, unsigned long i)
115 {
116 if (!os)
117 return NULL;
118 if (i >= os->used)
119 return NULL;
120 return os->os[i];
121 }
122
123 void obs_set_delete(ObsSet *os)
124 {
125 if (!os)
126 return;
127 if (os->os != NULL) {
128 unsigned long i = 0;
129 for (i = 0; i < os->used; i++) {
130 if (os->os[i]->key)
131 free(os->os[i]->key);
132 if (os->os[i]->inv_key)
133 free(os->os[i]->inv_key);
134 free(os->os[i]);
135 }
136 free(os->os);
137 }
138 free(os);
139 }
140
141 struct ObsDB {
142 rocksdb_t *db;
143 rocksdb_options_t *options;
144 rocksdb_writeoptions_t *writeoptions;
145 rocksdb_readoptions_t *readoptions;
146 rocksdb_mergeoperator_t *mergeop;
147 };
148
149 #define VALUE_LENGTH (sizeof(uint32_t) + (2 * sizeof(time_t)))
150
151 #define obsdb_max(a,b) \
152 ({ __typeof__ (a) _a = (a); \
153 __typeof__ (b) _b = (b); \
154 _a > _b ? _a : _b; })
155
156 #define obsdb_min(a,b) \
157 ({ __typeof__ (a) _a = (a); \
158 __typeof__ (b) _b = (b); \
159 _a < _b ? _a : _b; })
160
161 static inline int obs2buf(Observation *o, char **buf, size_t *buflen) {
162 uint32_t a,b,c;
163 int ret = 0;
164 tpl_node *tn = tpl_map("uuu", &a, &b, &c);
165 a = o->count;
166 b = o->last_seen;
167 c = o->first_seen;
168 ret = tpl_pack(tn, 0);
169 if (ret == 0) {
170 ret = tpl_dump(tn, TPL_MEM, buf, buflen);
171 }
172 tpl_free(tn);
173 return ret;
174 }
175
176 static inline int buf2obs(Observation *o, const char *buf, size_t buflen) {
177 uint32_t a,b,c;
178 int ret = 0;
179 tpl_node *tn = tpl_map("uuu", &a, &b, &c);
180 ret = tpl_load(tn, TPL_MEM, buf, buflen);
181 if (ret == 0) {
182 (void) tpl_unpack(tn, 0);
183 o->count = a;
184 o->last_seen = b;
185 o->first_seen = c;
186 }
187 tpl_free(tn);
188 return ret;
189 }
190
191 static char* obsdb_mergeop_full_merge(void *state, const char* key,
192 size_t key_length, const char* existing_value,
193 size_t existing_value_length,
194 const char* const* operands_list,
195 const size_t* operands_list_length,
196 int num_operands, unsigned char* success,
197 size_t* new_value_length)
198 {
199 Observation obs = {NULL, NULL, 0, 0, 0};
200
201 if (key_length < 1) {
202 fprintf(stderr, "full merge: key too short\n");
203 *success = (unsigned char) 0;
204 return NULL;
205 }
206 if (key[0] == 'i') {
207 /* this is an inverted index key with no meaningful value */
208 char *res = malloc(1 * sizeof(char));
209 *res = '\0';
210 *new_value_length = 1;
211 *success = 1;
212 return res;
213 } else if (key[0] == 'o') {
214 /* this is an observation value */
215 int i;
216 size_t buflength;
217 char *buf = NULL;
218 if (existing_value) {
219 buf2obs(&obs, existing_value, existing_value_length);
220 }
221 for (i = 0; i < num_operands; i++) {
222 Observation nobs = {NULL, NULL, 0, 0, 0};
223 buf2obs(&nobs, operands_list[i], operands_list_length[i]);
224 if (i == 0) {
225 if (!existing_value) {
226 obs.count = nobs.count;
227 obs.last_seen = nobs.last_seen;
228 obs.first_seen = nobs.first_seen;
229 } else {
230 obs.count += nobs.count;
231 obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
232 obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
233 }
234 } else {
235 obs.count += nobs.count;
236 obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
237 obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
238 }
239 }
240 obs2buf(&obs, &buf, &buflength);
241 *new_value_length = buflength;
242 *success = (unsigned char) 1;
243 return buf;
244 } else {
245 /* weird key format! */
246 fprintf(stderr, "full merge: weird key format\n");
247 *success = (unsigned char) 0;
248 return NULL;
249 }
250 }
251
252 static char* obsdb_mergeop_partial_merge(void *state, const char* key,
253 size_t key_length,
254 const char* const* operands_list,
255 const size_t* operands_list_length,
256 int num_operands, unsigned char* success,
257 size_t* new_value_length)
258 {
259 Observation obs = {NULL, NULL, 0, 0, 0};
260
261 if (key_length < 1) {
262 fprintf(stderr, "partial merge: key too short\n");
263 *success = (unsigned char) 0;
264 return NULL;
265 }
266 if (key[0] == 'i') {
267 /* this is an inverted index key with no meaningful value */
268 char *res = malloc(1 * sizeof(char));
269 *res = '\0';
270 *new_value_length = 1;
271 *success = 1;
272 return res;
273 } else if (key[0] == 'o') {
274 /* this is an observation value */
275 int i;
276 size_t buflength;
277 char *buf = NULL;
278 for (i = 0; i < num_operands; i++) {
279 Observation nobs = {NULL, NULL, 0, 0, 0};
280 buf2obs(&nobs, operands_list[i], operands_list_length[i]);
281 if (i == 0) {
282 obs.count = nobs.count;
283 obs.last_seen = nobs.last_seen;
284 obs.first_seen = nobs.first_seen;
285 } else {
286 obs.count += nobs.count;
287 obs.last_seen = obsdb_max(obs.last_seen, nobs.last_seen);
288 obs.first_seen = obsdb_min(obs.first_seen, nobs.first_seen);
289 }
290 }
291 obs2buf(&obs, &buf, &buflength);
292 *new_value_length = buflength;
293 *success = (unsigned char) 1;
294 return buf;
295 } else {
296 /* weird key format! */
297 fprintf(stderr, "partial merge: weird key format\n");
298 *success = (unsigned char) 0;
299 return NULL;
300 }
301 }
302
303 static void obsdb_mergeop_destructor(void *state)
304 {
305 return;
306 }
307
308 static const char* obsdb_mergeop_name(void *state)
309 {
310 return "observation mergeop";
311 }
312
313 static ObsDB* _obsdb_open(const char *path, size_t membudget, Error *e, bool readonly)
314 {
315 char *err = NULL;
316 int level_compression[5] = {
317 rocksdb_lz4_compression,
318 rocksdb_lz4_compression,
319 rocksdb_lz4_compression,
320 rocksdb_lz4_compression,
321 rocksdb_lz4_compression
322 };
323 ObsDB *db = calloc(1, sizeof(ObsDB));
324 if (db == NULL) {
325 if (e)
326 error_set(e, "could not allocate memory");
327 return NULL;
328 }
329
330 db->mergeop = rocksdb_mergeoperator_create(NULL,
331 obsdb_mergeop_destructor,
332 obsdb_mergeop_full_merge,
333 obsdb_mergeop_partial_merge,
334 NULL,
335 obsdb_mergeop_name);
336
337 db->options = rocksdb_options_create();
338 rocksdb_options_increase_parallelism(db->options, 8);
339 if (!readonly)
340 rocksdb_options_optimize_level_style_compaction(db->options, membudget);
341 rocksdb_options_set_create_if_missing(db->options, 1);
342 rocksdb_options_set_max_log_file_size(db->options, 10*1024*1024);
343 rocksdb_options_set_keep_log_file_num(db->options, 2);
344 rocksdb_options_set_max_open_files(db->options, 300);
345 rocksdb_options_set_merge_operator(db->options, db->mergeop);
346 rocksdb_options_set_compression_per_level(db->options, level_compression, 5);
347
348 if (!readonly)
349 db->db = rocksdb_open(db->options, path, &err);
350 else
351 db->db = rocksdb_open_for_read_only(db->options, path, 0, &err);
352 if (err) {
353 if (e)
354 error_set(e, err);
355 free(err);
356 return NULL;
357 }
358
359 db->writeoptions = rocksdb_writeoptions_create();
360 db->readoptions = rocksdb_readoptions_create();
361
362 return db;
363 }
364
365 ObsDB* obsdb_open(const char *path, size_t membudget, Error *e) {
366 return _obsdb_open(path, membudget, e, false);
367 }
368
369 ObsDB* obsdb_open_readonly(const char *path, Error *e) {
370 return _obsdb_open(path, 0, e, true);
371 }
372
373 int obsdb_put(ObsDB *db, Observation *obs, Error *e)
374 {
375 char *err = NULL;
376 size_t buflength;
377 char *buf;
378 if (!db)
379 return -1;
380
381 (void) obs2buf(obs, &buf, &buflength);
382
383 rocksdb_merge(db->db, db->writeoptions, obs->key, strlen(obs->key),
384 buf, buflength, &err);
385 if (err) {
386 if (e)
387 error_set(e, err);
388 free(err);
389 free(buf);
390 return -1;
391
392 }
393 free(buf);
394
395 rocksdb_merge(db->db, db->writeoptions, obs->inv_key, strlen(obs->key),
396 "", 0, &err);
397 if (err) {
398 if (e)
399 error_set(e, err);
400 free(err);
401 return -1;
402 }
403
404 return 0;
405 }
406
407 int obsdb_dump(ObsDB *db, Error *e)
408 {
409 rocksdb_iterator_t *it;
410 if (!db)
411 return -1;
412
413 it = rocksdb_create_iterator(db->db, db->readoptions);
414 for (rocksdb_iter_seek(it, "o", 1);
415 rocksdb_iter_valid(it) != (unsigned char) 0;
416 rocksdb_iter_next(it)) {
417 size_t size = 0;
418 int ret = 0;
419 Observation *o = NULL;
420 const char *rkey = NULL, *val = NULL;
421
422 rkey = rocksdb_iter_key(it, &size);
423 o = calloc(1, sizeof(Observation));
424 if (!o) {
425 return -1;
426 }
427 o->key = calloc(size + 1, sizeof(char));
428 if (!o->key) {
429 free(o);
430 return -1;
431 }
432 strncpy(o->key, rkey, size);
433 o->key[size] = '\0';
434 val = rocksdb_iter_value(it, &size);
435 ret = buf2obs(o, val, size);
436 if (ret != 0) {
437 fprintf(stderr, "error\n");
438 }
439 cgoObsDump(o);
440 free(o->key);
441 free(o);
442 }
443 rocksdb_iter_destroy(it);
444
445 return 0;
446 }
447
448 ObsSet* obsdb_search(ObsDB *db, const char *qrdata, const char *qrrname,
449 const char *qrrtype, const char *qsensorID)
450 {
451 rocksdb_iterator_t *it;
452 ObsSet *os;
453 if (!db)
454 return NULL;
455
456 os = obs_set_create(OBS_SET_STEP_SIZE);
457
458 if (qrrname != NULL) {
459 char *prefix = NULL;
460 size_t prefixlen = 0;
461 if (qsensorID != NULL) {
462 prefixlen = strlen(qsensorID) + strlen(qrrname) + 4;
463 prefix = calloc(prefixlen, sizeof(char));
464 if (!prefix) {
465 obs_set_delete(os);
466 return NULL;
467 }
468 (void) snprintf(prefix, prefixlen, "o%c%s%c%s%c", 0x1f, qrrname, 0x1f, qsensorID, 0x1f);
469 } else {
470 prefixlen = strlen(qrrname) + 3;
471 prefix = calloc(prefixlen, sizeof(char));
472 if (!prefix) {
473 obs_set_delete(os);
474 return NULL;
475 }
476 (void) snprintf(prefix, prefixlen, "o%c%s%c", 0x1f, qrrname, 0x1f);
477 }
478 cgoLogDebug(prefix);
479
480 it = rocksdb_create_iterator(db->db, db->readoptions);
481 rocksdb_iter_seek(it, prefix, prefixlen);
482 for (; rocksdb_iter_valid(it) != (unsigned char) 0; rocksdb_iter_next(it)) {
483 size_t size = 0;
484 int ret = 0;
485 Observation *o = NULL;
486 const char *rkey = rocksdb_iter_key(it, &size), *val = NULL;
487 char *rrname = NULL, *sensorID = NULL, *rrtype = NULL, *rdata = NULL, *saveptr;
488 char *tokkey = calloc(size + 1, sizeof(char));
489 if (!tokkey) {
490 obs_set_delete(os);
491 rocksdb_iter_destroy(it);
492 free(prefix);
493 return NULL;
494 }
495
496 strncpy(tokkey, rkey, size);
497 tokkey[size] = '\0';
498 rrname = strtok_r(tokkey+2, "\x1f", &saveptr);
499 sensorID = strtok_r(NULL, "\x1f", &saveptr);
500 rrtype = strtok_r(NULL, "\x1f", &saveptr);
501 rdata = strtok_r(NULL, "\x1f", &saveptr);
502 if (rrname == NULL || (strcmp(rrname, qrrname) != 0)) {
503 free(tokkey);
504 tokkey = NULL;
505 break;
506 }
507 if (sensorID == NULL || (qsensorID != NULL && strcmp(qsensorID, sensorID) != 0)) {
508 free(tokkey);
509 tokkey = NULL;
510 continue;
511 }
512 if (rdata == NULL || (qrdata != NULL && strcmp(qrdata, rdata) != 0)) {
513 free(tokkey);
514 tokkey = NULL;
515 continue;
516 }
517 if (rrtype == NULL || (qrrtype != NULL && strcmp(qrrtype, rrtype) != 0)) {
518 free(tokkey);
519 tokkey = NULL;
520 continue;
521 }
522
523 o = calloc(1, sizeof(Observation));
524 if (!o) {
525 free(tokkey);
526 obs_set_delete(os);
527 rocksdb_iter_destroy(it);
528 free(prefix);
529 return NULL;
530 }
531 o->key = calloc(size + 1, sizeof(char));
532 if (!o->key) {
533 free(o);
534 free(tokkey);
535 obs_set_delete(os);
536 rocksdb_iter_destroy(it);
537 free(prefix);
538 return NULL;
539 }
540 strncpy(o->key, rkey, size);
541 o->key[size] = '\0';
542 val = rocksdb_iter_value(it, &size);
543 ret = buf2obs(o, val, size);
544 if (ret == 0) {
545 obs_set_add(os, o);
546 }
547 free(tokkey);
548 tokkey = NULL;
549 }
550 rocksdb_iter_destroy(it);
551 free(prefix);
552 } else {
553 char *prefix = NULL;
554 size_t prefixlen = 0;
555 if (qsensorID != NULL) {
556 prefixlen = strlen(qsensorID) + strlen(qrdata) + 4;
557 prefix = calloc(prefixlen, sizeof(char));
558 if (!prefix) {
559 obs_set_delete(os);
560 return NULL;
561 }
562 (void) snprintf(prefix, prefixlen, "i%c%s%c%s%c", 0x1f, qrdata, 0x1f, qsensorID, 0x1f);
563 } else {
564 prefixlen = strlen(qrdata) + 3;
565 prefix = calloc(prefixlen, sizeof(char));
566 if (!prefix) {
567 obs_set_delete(os);
568 return NULL;
569 }
570 (void) snprintf(prefix, prefixlen, "i%c%s%c", 0x1f, qrdata, 0x1f);
571 }
572 cgoLogDebug(prefix);
573
574 it = rocksdb_create_iterator(db->db, db->readoptions);
575 rocksdb_iter_seek(it, prefix, strlen(prefix));
576 for (; rocksdb_iter_valid(it) != (unsigned char) 0; rocksdb_iter_next(it)) {
577 size_t size = 0, fullkeylen;
578 int ret;
579 const char *rkey = rocksdb_iter_key(it, &size);
580 char *val = NULL;
581 char *rrname = NULL, *sensorID = NULL, *rrtype = NULL, *rdata = NULL, *saveptr;
582 char fullkey[4096];
583 char *err = NULL;
584 Observation *o = NULL;
585 char *tokkey = calloc(size + 1, sizeof(char));
586 if (!tokkey) {
587 obs_set_delete(os);
588 rocksdb_iter_destroy(it);
589 free(prefix);
590 return NULL;
591 }
592
593 strncpy(tokkey, rkey, size);
594 tokkey[size] = '\0';
595 rdata = strtok_r(tokkey+2, "\x1f", &saveptr);
596 sensorID = strtok_r(NULL, "\x1f", &saveptr);
597 rrname = strtok_r(NULL, "\x1f", &saveptr);
598 rrtype = strtok_r(NULL, "\x1f", &saveptr);
599 if (strcmp(rdata, qrdata) != 0) {
600 free(tokkey);
601 tokkey = NULL;
602 break;
603 }
604 cgoLogDebug(rdata);
605 if (sensorID == NULL || (qsensorID != NULL && strcmp(qsensorID, sensorID) != 0)) {
606 free(tokkey);
607 tokkey = NULL;
608 continue;
609 }
610 if (rdata == NULL || (qrdata != NULL && strcmp(qrdata, rdata) != 0)) {
611 free(tokkey);
612 tokkey = NULL;
613 continue;
614 }
615 if (rrtype == NULL || (qrrtype != NULL && strcmp(qrrtype, rrtype) != 0)) {
616 free(tokkey);
617 tokkey = NULL;
618 continue;
619 }
620
621 (void) snprintf(fullkey, 4096, "o%c%s%c%s%c%s%c%s", 0x1f, rrname, 0x1f, sensorID, 0x1f, rrtype, 0x1f, rdata);
622 cgoLogDebug(fullkey);
623
624 fullkeylen = strlen(fullkey);
625 val = rocksdb_get(db->db, db->readoptions, fullkey, fullkeylen, &size, &err);
626 if (err != NULL) {
627 cgoLogDebug("observation not found");
628 free(tokkey);
629 tokkey = NULL;
630 continue;
631 }
632
633 o = calloc(1, sizeof(Observation));
634 if (!o) {
635 free(tokkey);
636 obs_set_delete(os);
637 rocksdb_iter_destroy(it);
638 free(prefix);
639 free(val);
640 return NULL;
641 }
642 o->key = calloc(fullkeylen + 1, sizeof(char));
643 if (!o->key) {
644 free(o);
645 free(tokkey);
646 obs_set_delete(os);
647 rocksdb_iter_destroy(it);
648 free(prefix);
649 free(val);
650 return NULL;
651 }
652 strncpy(o->key, fullkey, fullkeylen);
653 o->key[fullkeylen] = '\0';
654 ret = buf2obs(o, val, size);
655 if (ret == 0) {
656 obs_set_add(os, o);
657 }
658 free(tokkey);
659 free(val);
660 tokkey = NULL;
661 }
662 rocksdb_iter_destroy(it);
663 free(prefix);
664 }
665
666 return os;
667 }
668
669 unsigned long obsdb_num_keys(ObsDB *db)
670 {
671 const char *val;
672 if (!db)
673 return 0;
674 val = rocksdb_property_value(db->db, "rocksdb.estimate-num-keys");
675 if (!val) {
676 return 0;
677 } else {
678 unsigned long v;
679 int ret;
680 ret = sscanf(val, "%lu", &v);
681 if (ret != 1) {
682 return 0;
683 } else {
684 return v;
685 }
686 }
687 }
688
689 void obsdb_close(ObsDB *db)
690 {
691 if (!db)
692 return;
693 rocksdb_mergeoperator_destroy(db->mergeop);
694 rocksdb_writeoptions_destroy(db->writeoptions);
695 rocksdb_readoptions_destroy(db->readoptions);
696 // rocksdb_options_destroy(db->options);
697 rocksdb_close(db->db);
698 }
+46
-0
db/obs_rocksdb.h less more
0 /*
1 balboa
2 Copyright (c) 2018, DCSO GmbH
3 */
4
5 #ifndef OBS_ROCKSDB_H
6 #define OBS_ROCKSDB_H
7
8 #include <stdbool.h>
9 #include <stdint.h>
10 #include <time.h>
11
12 typedef struct Error Error;
13
14 Error* error_new();
15 const char* error_get(Error*);
16 bool error_is_set(Error*);
17 void error_delete(Error*);
18
19 typedef struct {
20 char *key,
21 *inv_key;
22 uint32_t count,
23 last_seen,
24 first_seen;
25 } Observation;
26
27 typedef struct ObsSet ObsSet;
28
29 unsigned long obs_set_size(ObsSet*);
30 const Observation* obs_set_get(ObsSet*, unsigned long);
31 void obs_set_delete(ObsSet*);
32
33 typedef struct ObsDB ObsDB;
34
35 ObsDB* obsdb_open(const char *path, size_t membudget, Error*);
36 ObsDB* obsdb_open_readonly(const char *path, Error*);
37 int obsdb_put(ObsDB *db, Observation *obs, Error*);
38 ObsSet* obsdb_search(ObsDB *db, const char *qrdata, const char *qrrname,
39 const char *qrrtype, const char *qsensorID);
40 int obsdb_dump(ObsDB *db, Error *e) ;
41
42 unsigned long obsdb_num_keys(ObsDB*);
43 void obsdb_close(ObsDB*);
44
45 #endif⏎
+64
-0
doc/cassandra_schema.txt less more
0 CREATE KEYSPACE IF NOT EXISTS balboa WITH REPLICATION = {
1 'class' : 'SimpleStrategy',
2 'replication_factor' : 2
3 };
4
5 CREATE TABLE balboa.observations_by_rdata (
6 rrname text,
7 rdata text,
8 rrtype text,
9 sensor_id text,
10 last_seen timestamp,
11 PRIMARY KEY (rdata, rrname, rrtype, sensor_id)
12 );
13 CREATE CUSTOM INDEX ON balboa.observations_by_rdata (rrname) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
14 'analyzed' : 'true',
15 'analyzer_class' : 'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
16 'case_sensitive' : 'false',
17 'mode' : 'CONTAINS'
18 };
19 ALTER TABLE balboa.observations_by_rdata
20 WITH COMPRESSION = {'sstable_compression': 'LZ4Compressor',
21 'chunk_length_kb': 64};
22
23 CREATE TABLE balboa.observations_by_rrname (
24 rrname text,
25 rdata text,
26 rrtype text,
27 sensor_id text,
28 last_seen timestamp,
29 PRIMARY KEY (rrname, rdata, rrtype, sensor_id)
30 );
31 CREATE CUSTOM INDEX ON balboa.observations_by_rrname (rdata) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
32 'analyzed' : 'true',
33 'analyzer_class' : 'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
34 'case_sensitive' : 'false',
35 'mode' : 'CONTAINS'
36 };
37 ALTER TABLE balboa.observations_by_rrname
38 WITH COMPRESSION = {'sstable_compression': 'LZ4Compressor',
39 'chunk_length_kb': 64};
40
41 CREATE TABLE balboa.observations_counts (
42 rrname text,
43 rdata text,
44 rrtype text,
45 sensor_id text,
46 count counter,
47 PRIMARY KEY (rrname, rdata, rrtype, sensor_id)
48 );
49 ALTER TABLE balboa.observations_counts
50 WITH COMPRESSION = {'sstable_compression': 'LZ4Compressor',
51 'chunk_length_kb': 64};
52
53 CREATE TABLE balboa.observations_firstseen (
54 rrname text,
55 rdata text,
56 rrtype text,
57 sensor_id text,
58 first_seen timestamp,
59 PRIMARY KEY (rrname, rdata, rrtype, sensor_id)
60 );
61 ALTER TABLE balboa.observations_firstseen
62 WITH COMPRESSION = {'sstable_compression': 'LZ4Compressor',
63 'chunk_length_kb': 64};
+15
-0
doc/gamelinux_config.txt less more
0 input {
1 file {
2 path => "/var/log/passivedns.log"
3 }
4 }
5
6 output {
7 http {
8 http_method => 'post'
9 url => 'http://localhost:8081/submit'
10 format => "message"
11 message => "%{message}"
12 headers => ["X-Sensor-ID", "abcde"]
13 }
14 }
+37
-0
doc/packetbeat_config.txt less more
0 >>> /etc/packetbeat/packetbeat.yml
1 ----------------------------------
2
3 packetbeat.interfaces.device: any
4
5 packetbeat.protocols:
6 - type: dns
7 ports: [53]
8 include_authorities: true
9 include_additionals: true
10
11 output.logstash:
12 hosts: ["localhost:5044"]
13
14
15 >>> /etc/logstash/conf.d/pdns.conf
16 ----------------------------------
17
18 input {
19 beats {
20 port => 5044
21 }
22 }
23
24 filter {
25 mutate {
26 add_field => { "sensor_id" => "abcde" }
27 }
28 }
29
30 output {
31 http {
32 http_method => 'post'
33 url => 'http://localhost:8081/submit'
34 headers => ["X-Sensor-ID", "abcde"]
35 }
36 }
+20
-0
feeder/feeder.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package feeder
4
5 import (
6 "github.com/DCSO/balboa/format"
7 "github.com/DCSO/balboa/observation"
8 )
9
10 // Feeder is an interface of a component that accepts observations in a
11 // specific format and feeds them into a channel of InputObservations.
12 // An input decoder in the form of a MakeObservationFunc describes the
13 // operations necessary to transform the input format into an
14 // InputObservation.
15 type Feeder interface {
16 Run(chan observation.InputObservation) error
17 SetInputDecoder(format.MakeObservationFunc)
18 Stop(chan bool)
19 }
+339
-0
feeder/feeder_amqp.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package feeder
4
5 import (
6 "bytes"
7 "compress/gzip"
8 "fmt"
9 "io"
10 "strings"
11 "sync"
12 "time"
13
14 "github.com/DCSO/balboa/format"
15 "github.com/DCSO/balboa/observation"
16
17 "github.com/NeowayLabs/wabbit"
18 amqp "github.com/NeowayLabs/wabbit/amqp"
19 log "github.com/sirupsen/logrus"
20 origamqp "github.com/streadway/amqp"
21 )
22
23 // Consumer reads and processes messages from a fake RabbitMQ server.
24 type Consumer struct {
25 conn wabbit.Conn
26 channel wabbit.Channel
27 tag string
28 exchanges []string
29 URL string
30 done chan error
31 stop chan bool
32 deliveries <-chan wabbit.Delivery
33 Callback func(wabbit.Delivery)
34 StopReconnection chan bool
35 ChanMutex sync.Mutex
36 ConnMutex sync.Mutex
37 OutChan chan observation.InputObservation
38 MakeObservationFunc format.MakeObservationFunc
39 ErrorChan chan wabbit.Error
40 Reconnector func(string) (wabbit.Conn, string, error)
41 Connector func(*Consumer) error
42 }
43
44 func decompressBody(d wabbit.Delivery) ([]byte, error) {
45 var compressedBuffer bytes.Buffer
46 compressedBuffer.Write(d.Body())
47 compressedReader, err := gzip.NewReader(&compressedBuffer)
48 if err != nil {
49 return nil, err
50 }
51 decompressedWriter := new(bytes.Buffer)
52 io.Copy(decompressedWriter, compressedReader)
53 compressedReader.Close()
54 body := decompressedWriter.Bytes()
55 return body, nil
56 }
57
58 const amqpReconnDelay = 2 * time.Second
59
60 func reconnectOnFailure(s *Consumer) {
61 for {
62 select {
63 case <-s.StopReconnection:
64 return
65 case rabbitErr := <-s.ErrorChan:
66 if rabbitErr != nil {
67 log.Warnf("RabbitMQ connection failed: %s", rabbitErr.Reason())
68 close(s.stop)
69 for {
70 time.Sleep(amqpReconnDelay)
71 connErr := s.Connector(s)
72 if connErr != nil {
73 log.Warnf("RabbitMQ error: %s", connErr)
74 } else {
75 log.Infof("Reestablished connection to %s", s.URL)
76 s.ConnMutex.Lock()
77 s.conn.NotifyClose(s.ErrorChan)
78 s.ConnMutex.Unlock()
79 go handle(s.deliveries, s.stop, s.done, s.OutChan, s.MakeObservationFunc)
80 break
81 }
82 }
83 }
84 }
85 }
86 }
87
88 // NewConsumerWithReconnector creates a new consumer with the given properties. The callback
89 // function is called for each delivery accepted from a consumer channel.
90 func (f *AMQPFeeder) NewConsumerWithReconnector(amqpURI string, exchanges []string, exchangeType,
91 queueName, key, ctag string, out chan observation.InputObservation,
92 reconnector func(string) (wabbit.Conn, string, error)) (*Consumer, error) {
93 var err error
94 c := &Consumer{
95 conn: nil,
96 channel: nil,
97 exchanges: exchanges,
98 URL: amqpURI,
99 tag: ctag,
100 done: make(chan error),
101 stop: make(chan bool),
102 OutChan: out,
103 MakeObservationFunc: f.MakeObservationFunc,
104 Reconnector: reconnector,
105 StopReconnection: make(chan bool),
106 }
107
108 c.Connector = func(s *Consumer) error {
109 var err error
110 var exchangeType string
111
112 s.ConnMutex.Lock()
113 s.conn, exchangeType, err = s.Reconnector(s.URL)
114 s.ConnMutex.Unlock()
115 if err != nil {
116 return err
117 }
118 s.ChanMutex.Lock()
119 s.channel, err = s.conn.Channel()
120 s.ChanMutex.Unlock()
121 if err != nil {
122 s.ConnMutex.Lock()
123 s.conn.Close()
124 s.ConnMutex.Unlock()
125 return err
126 }
127
128 for _, exchange := range exchanges {
129 // We do not want to declare an exchange on non-default connection methods,
130 // as they may not support all exchange types. For instance amqptest does
131 // not support 'fanout'.
132 log.Debug("declaring exchange ", exchange)
133 err = s.channel.ExchangeDeclare(
134 exchange,
135 exchangeType,
136 wabbit.Option{
137 "durable": true,
138 "autoDelete": false,
139 "internal": false,
140 "noWait": false,
141 },
142 )
143 if err != nil {
144 log.Error(err)
145 s.ChanMutex.Lock()
146 s.channel.Close()
147 s.ChanMutex.Unlock()
148 s.ConnMutex.Lock()
149 s.conn.Close()
150 s.ConnMutex.Unlock()
151 return err
152 }
153 }
154 queueName := fmt.Sprintf("%s.%s", strings.Join(exchanges, "."), queueName)
155 queue, err := c.channel.QueueDeclare(
156 queueName,
157 wabbit.Option{
158 "durable": false,
159 "autoDelete": true,
160 "exclusive": true,
161 "noWait": false,
162 "args": origamqp.Table{
163 "x-message-ttl": int32(300000),
164 "x-max-length-bytes": int32(100 * 1024 * 1024),
165 },
166 },
167 )
168 if err != nil {
169 return fmt.Errorf("Queue Declare: %s", err)
170 }
171 log.Debugf("declared Queue (%q %d messages, %d consumers), binding to Exchange (key %q)",
172 queue.Name(), queue.Messages(), queue.Consumers(), key)
173
174 for _, exchange := range exchanges {
175 log.Debug("binding to exchange ", exchange)
176 if err = c.channel.QueueBind(
177 queue.Name(),
178 key,
179 exchange,
180 wabbit.Option{
181 "noWait": false,
182 },
183 ); err != nil {
184 return fmt.Errorf("Queue Bind: %s", err)
185 }
186 }
187
188 log.Debugf("Queue bound to Exchange, starting Consume (consumer tag %q)", c.tag)
189 c.deliveries, err = c.channel.Consume(
190 queue.Name(),
191 c.tag,
192 wabbit.Option{
193 "autoAck": false,
194 "exclusive": false,
195 "noLocal": false,
196 "noWait": false,
197 },
198 )
199 if err != nil {
200 return fmt.Errorf("Queue Consume: %s", err)
201 }
202
203 log.Debugf("Consumer established connection to %s", s.URL)
204 s.stop = make(chan bool)
205 c.ErrorChan = make(chan wabbit.Error)
206
207 return nil
208 }
209
210 c.ErrorChan = make(chan wabbit.Error)
211 err = c.Connector(c)
212 if err != nil {
213 return nil, err
214 }
215 c.conn.NotifyClose(c.ErrorChan)
216
217 go reconnectOnFailure(c)
218 go handle(c.deliveries, c.stop, c.done, out, f.MakeObservationFunc)
219
220 return c, nil
221 }
222
223 func defaultReconnector(amqpURI string) (wabbit.Conn, string, error) {
224 conn, err := amqp.Dial(amqpURI)
225 if err != nil {
226 return nil, "fanout", err
227 }
228 return conn, "fanout", err
229 }
230
231 // NewConsumer returns a new Consumer.
232 func (f *AMQPFeeder) NewConsumer(amqpURI string, exchanges []string, exchangeType, queueName, key,
233 ctag string, out chan observation.InputObservation) (*Consumer, error) {
234 return f.NewConsumerWithReconnector(amqpURI, exchanges, exchangeType, queueName, key,
235 ctag, out, defaultReconnector)
236 }
237
238 // Shutdown shuts down a consumer, closing down its channels and connections.
239 func (c *Consumer) Shutdown() error {
240 // will close() the deliveries channel
241 if err := c.channel.Close(); err != nil {
242 return fmt.Errorf("Channel close failed: %s", err)
243 }
244 if err := c.conn.Close(); err != nil {
245 return fmt.Errorf("AMQP connection close error: %s", err)
246 }
247 defer log.Debugf("AMQP shutdown OK")
248 // wait for handle() to exit
249 return <-c.done
250 }
251
252 func handle(deliveries <-chan wabbit.Delivery, stop chan bool, done chan error,
253 out chan observation.InputObservation, fn format.MakeObservationFunc) {
254 for {
255 select {
256 case <-stop:
257 done <- nil
258 return
259 case d := <-deliveries:
260 if d == nil {
261 done <- nil
262 return
263 }
264 log.Infof("got %d bytes via AMQP", len(d.Body()))
265 raw := d.Body()
266 if _, ok := d.Headers()["compressed"]; ok {
267 var err error
268 raw, err = decompressBody(d)
269 if err != nil {
270 log.Warn(err)
271 continue
272 }
273 }
274 var sensorID string
275 if _, ok := d.Headers()["sensor_id"]; ok {
276 sensorID = d.Headers()["sensor_id"].(string)
277 }
278 err := fn(raw, sensorID, out, stop)
279 if err != nil {
280 log.Warn(err)
281 }
282 d.Ack(true)
283 }
284 }
285
286 }
287
288 // AMQPFeeder is a Feeder that accepts input via AMQP queues.
289 type AMQPFeeder struct {
290 StopChan chan bool
291 StoppedChan chan bool
292 IsRunning bool
293 Consumer *Consumer
294 URL string
295 Exchanges []string
296 Queue string
297 MakeObservationFunc format.MakeObservationFunc
298 }
299
300 // MakeAMQPFeeder returns a new AMQPFeeder, connecting to the AMQP server at
301 // the given URL, creating a new queue with the given name bound to the
302 // provided exchanges.
303 func MakeAMQPFeeder(url string, exchanges []string, queue string) *AMQPFeeder {
304 return &AMQPFeeder{
305 IsRunning: false,
306 StopChan: make(chan bool),
307 Exchanges: exchanges,
308 URL: url,
309 Queue: queue,
310 MakeObservationFunc: format.MakeFeverAggregateInputObservations,
311 }
312 }
313
314 // SetInputDecoder states that the given MakeObservationFunc should be used to
315 // parse and decode data delivered to this feeder.
316 func (f *AMQPFeeder) SetInputDecoder(fn format.MakeObservationFunc) {
317 f.MakeObservationFunc = fn
318 }
319
320 // Run starts the feeder.
321 func (f *AMQPFeeder) Run(out chan observation.InputObservation) error {
322 var err error
323 f.Consumer, err = f.NewConsumer(f.URL, f.Exchanges, "fanout", f.Queue, "", "balboa", out)
324 if err != nil {
325 return err
326 }
327 f.IsRunning = true
328 return nil
329 }
330
331 // Stop causes the feeder to stop accepting deliveries and close all
332 // associated channels, including the passed notification channel.
333 func (f *AMQPFeeder) Stop(stopChan chan bool) {
334 close(stopChan)
335 if f.IsRunning {
336 f.Consumer.Shutdown()
337 }
338 }
+133
-0
feeder/feeder_config.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package feeder
4
5 import (
6 "fmt"
7 "strings"
8
9 "github.com/DCSO/balboa/format"
10 "github.com/DCSO/balboa/observation"
11
12 log "github.com/sirupsen/logrus"
13 yaml "gopkg.in/yaml.v2"
14 )
15
16 // Setup describes a collection of feeders that should be active, including
17 // their configuration settings.
18 type Setup struct {
19 Feeder []struct {
20 Name string `yaml:"name"`
21 Type string `yaml:"type"`
22 InputFormat string `yaml:"input_format"`
23 // for AMQP
24 URL string `yaml:"url"`
25 Exchange []string `yaml:"exchange"`
26 // for HTTP etc.
27 ListenHost string `yaml:"listen_host"`
28 ListenPort int `yaml:"listen_port"`
29 // for socket input
30 Path string `yaml:"path"`
31 } `yaml:"feeder"`
32 Feeders map[string]Feeder
33 }
34
35 // LoadSetup creates a new Setup from a byte array containing YAML.
36 func LoadSetup(in []byte) (*Setup, error) {
37 var fs Setup
38 seenFeeders := make(map[string]bool)
39 err := yaml.Unmarshal(in, &fs)
40 if err != nil {
41 return nil, err
42 }
43 fs.Feeders = make(map[string]Feeder)
44 for _, f := range fs.Feeder {
45 if f.Name == "" {
46 return nil, fmt.Errorf("name missing")
47 }
48 if _, ok := seenFeeders[f.Name]; ok {
49 return nil, fmt.Errorf("duplicate name: %s", f.Name)
50 }
51 seenFeeders[f.Name] = true
52 if f.Type == "" {
53 return nil, fmt.Errorf("type missing")
54 }
55 if f.InputFormat == "" {
56 return nil, fmt.Errorf("input format missing")
57 }
58 switch f.Type {
59 case "amqp":
60 if len(f.Exchange) == 0 {
61 return nil, fmt.Errorf("%s: Exchange missing", f.Name)
62 }
63 if f.URL == "" {
64 return nil, fmt.Errorf("%s: URL missing", f.Name)
65 }
66 case "http":
67 if f.ListenHost == "" {
68 return nil, fmt.Errorf("%s: ListenHost missing", f.Name)
69 }
70 if f.ListenPort == 0 {
71 return nil, fmt.Errorf("%s: ListenPort missing", f.Name)
72 }
73 case "socket":
74 if f.Path == "" {
75 return nil, fmt.Errorf("%s: socket Path missing", f.Name)
76 }
77 }
78 }
79 return &fs, nil
80 }
81
82 // Run starts all feeders according to the description in the setup, in the
83 // background. Use Stop() to stop the feeders.
84 func (fs *Setup) Run(in chan observation.InputObservation) error {
85 for _, v := range fs.Feeder {
86 log.Infof("starting feeder %s", v.Name)
87 switch v.Type {
88 case "amqp":
89 queueName := strings.ToLower(strings.Replace(v.Name, " ", "_", -1))
90 fs.Feeders[v.Name] = MakeAMQPFeeder(v.URL, v.Exchange, queueName)
91 fs.Feeders[v.Name].Run(in)
92 case "http":
93 fs.Feeders[v.Name] = MakeHTTPFeeder(v.ListenHost, v.ListenPort)
94 fs.Feeders[v.Name].Run(in)
95 case "socket":
96 f, err := MakeSocketFeeder(v.Path)
97 if err != nil {
98 return err
99 }
100 fs.Feeders[v.Name] = f
101 fs.Feeders[v.Name].Run(in)
102 }
103 switch v.InputFormat {
104 case "fever_aggregate":
105 fs.Feeders[v.Name].SetInputDecoder(format.MakeFeverAggregateInputObservations)
106 case "gopassivedns":
107 fs.Feeders[v.Name].SetInputDecoder(format.MakeGopassivednsInputObservations)
108 case "packetbeat":
109 fs.Feeders[v.Name].SetInputDecoder(format.MakePacketbeatInputObservations)
110 case "suricata_dns":
111 fs.Feeders[v.Name].SetInputDecoder(format.MakeSuricataInputObservations)
112 case "gamelinux":
113 fs.Feeders[v.Name].SetInputDecoder(format.MakeFjellskaalInputObservations)
114 default:
115 log.Fatalf("unknown input format: %s", v.InputFormat)
116 }
117 }
118 return nil
119 }
120
121 // Stop causes all feeders described in the setup to cease processing input.
122 // The stopChan will be closed once all feeders are done shutting down.
123 func (fs *Setup) Stop(stopChan chan bool) {
124 for k, v := range fs.Feeders {
125 log.Infof("stopping feeder %s", k)
126 myStopChan := make(chan bool)
127 v.Stop(myStopChan)
128 <-myStopChan
129 log.Infof("feeder %s stopped", k)
130 }
131 close(stopChan)
132 }
+88
-0
feeder/feeder_http.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package feeder
4
5 import (
6 "context"
7 "fmt"
8 "io/ioutil"
9 "net/http"
10
11 "github.com/DCSO/balboa/format"
12 "github.com/DCSO/balboa/observation"
13
14 log "github.com/sirupsen/logrus"
15 )
16
17 // HTTPFeeder is a Feeder implementation that accepts HTTP requests to obtain
18 // observations.
19 type HTTPFeeder struct {
20 StopChan chan bool
21 StoppedChan chan bool
22 IsRunning bool
23 Port int
24 Host string
25 MakeObservationFunc format.MakeObservationFunc
26 Server *http.Server
27 OutChan chan observation.InputObservation
28 }
29
30 // MakeHTTPFeeder creates a new HTTPFeeder listening on a specific address
31 // and port.
32 func MakeHTTPFeeder(host string, port int) *HTTPFeeder {
33 return &HTTPFeeder{
34 IsRunning: false,
35 StopChan: make(chan bool),
36 Port: port,
37 Host: host,
38 MakeObservationFunc: format.MakeFeverAggregateInputObservations,
39 }
40 }
41
42 // SetInputDecoder states that the given MakeObservationFunc should be used to
43 // parse and decode data delivered to this feeder.
44 func (f *HTTPFeeder) SetInputDecoder(fn format.MakeObservationFunc) {
45 f.MakeObservationFunc = fn
46 }
47
48 func (f *HTTPFeeder) ServeHTTP(w http.ResponseWriter, r *http.Request) {
49 sensorID := r.Header.Get("X-Sensor-ID")
50 if r.Method == http.MethodPost {
51 body, err := ioutil.ReadAll(r.Body)
52 log.Infof("got %d bytes via HTTP", len(body))
53 if err != nil {
54 log.Warn(err)
55 return
56 }
57 f.MakeObservationFunc(body, sensorID, f.OutChan, f.StopChan)
58 }
59 w.WriteHeader(200)
60 }
61
62 // Run starts the feeder.
63 func (f *HTTPFeeder) Run(out chan observation.InputObservation) error {
64 f.OutChan = out
65 f.Server = &http.Server{
66 Addr: fmt.Sprintf("%s:%d", f.Host, f.Port),
67 Handler: f,
68 }
69 log.Infof("accepting submissions on port %v", f.Port)
70 go func() {
71 err := f.Server.ListenAndServe()
72 if err != nil {
73 log.Info(err)
74 }
75 }()
76 f.IsRunning = true
77 return nil
78 }
79
80 // Stop causes the feeder to stop accepting requests and close all
81 // associated channels, including the passed notification channel.
82 func (f *HTTPFeeder) Stop(stopChan chan bool) {
83 if f.IsRunning {
84 f.Server.Shutdown(context.TODO())
85 }
86 close(stopChan)
87 }
+116
-0
feeder/feeder_socket.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package feeder
4
5 import (
6 "bufio"
7 "net"
8 "time"
9
10 "github.com/DCSO/balboa/format"
11 "github.com/DCSO/balboa/observation"
12
13 log "github.com/sirupsen/logrus"
14 )
15
16 // SocketFeeder is a Feeder implementation that reds data from a UNIX socket.
17 type SocketFeeder struct {
18 ObsChan chan observation.InputObservation
19 Verbose bool
20 Running bool
21 InputListener net.Listener
22 MakeObservationFunc format.MakeObservationFunc
23 StopChan chan bool
24 StoppedChan chan bool
25 }
26
27 func (sf *SocketFeeder) handleServerConnection() {
28 for {
29 select {
30 case <-sf.StopChan:
31 sf.InputListener.Close()
32 close(sf.StoppedChan)
33 return
34 default:
35 sf.InputListener.(*net.UnixListener).SetDeadline(time.Now().Add(1e9))
36 c, err := sf.InputListener.Accept()
37 if nil != err {
38 if opErr, ok := err.(*net.OpError); ok && opErr.Timeout() {
39 continue
40 }
41 log.Info(err)
42 }
43
44 scanner := bufio.NewScanner(c)
45 buf := make([]byte, 0, 32*1024*1024)
46 scanner.Buffer(buf, 32*1024*1024)
47 for {
48 for scanner.Scan() {
49 select {
50 case <-sf.StopChan:
51 sf.InputListener.Close()
52 close(sf.StoppedChan)
53 return
54 default:
55 json := scanner.Bytes()
56 sf.MakeObservationFunc(json, "[unknown]", sf.ObsChan, sf.StopChan)
57 }
58 }
59 errRead := scanner.Err()
60 if errRead == nil {
61 break
62 } else if errRead == bufio.ErrTooLong {
63 log.Warn(errRead)
64 scanner = bufio.NewScanner(c)
65 scanner.Buffer(buf, 2*cap(buf))
66 } else {
67 log.Warn(errRead)
68 }
69 }
70 }
71 }
72 }
73
74 // MakeSocketFeeder returns a new SocketFeeder reading from the Unix socket
75 // inputSocket and writing parsed events to outChan. If no such socket could be
76 // created for listening, the error returned is set accordingly.
77 func MakeSocketFeeder(inputSocket string) (*SocketFeeder, error) {
78 var err error
79 si := &SocketFeeder{
80 Verbose: false,
81 StopChan: make(chan bool),
82 }
83 si.InputListener, err = net.Listen("unix", inputSocket)
84 if err != nil {
85 return nil, err
86 }
87 return si, err
88 }
89
90 // SetInputDecoder states that the given MakeObservationFunc should be used to
91 // parse and decode data delivered to this feeder.
92 func (sf *SocketFeeder) SetInputDecoder(fn format.MakeObservationFunc) {
93 sf.MakeObservationFunc = fn
94 }
95
96 // Run starts the feeder.
97 func (sf *SocketFeeder) Run(out chan observation.InputObservation) error {
98 if !sf.Running {
99 sf.ObsChan = out
100 sf.Running = true
101 sf.StopChan = make(chan bool)
102 go sf.handleServerConnection()
103 }
104 return nil
105 }
106
107 // Stop causes the SocketFeeder to stop reading from the socket and close all
108 // associated channels, including the passed notification channel.
109 func (sf *SocketFeeder) Stop(stoppedChan chan bool) {
110 if sf.Running {
111 sf.StoppedChan = stoppedChan
112 close(sf.StopChan)
113 sf.Running = false
114 }
115 }
+19
-0
feeders.yaml less more
0 feeder:
1 - name: AMQPInput2
2 type: amqp
3 url: amqp://guest:guest@localhost:5672
4 exchange: [ tdh.pdns ]
5 input_format: fever_aggregate
6 - name: HTTP Input
7 type: http
8 listen_host: 127.0.0.1
9 listen_port: 8081
10 input_format: fever_aggregate
11 - name: Socket Input
12 type: socket
13 path: /tmp/balboa.sock
14 input_format: fever_aggregate
15 - name: Suricata Socket Input
16 type: socket
17 path: /tmp/suri.sock
18 input_format: suricata_dns
+11
-0
format/format.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import "github.com/DCSO/balboa/observation"
6
7 // MakeObservationFunc is a function that accepts a byte array with input
8 // obtained from a feeder, a sensor ID, a channel for the generated
9 // InputObservations, and a channel to signal a stop.
10 type MakeObservationFunc func([]byte, string, chan observation.InputObservation, chan bool) error
+69
-0
format/format_fever.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "encoding/json"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10
11 log "github.com/sirupsen/logrus"
12 )
13
14 type rdata struct {
15 AnsweringHost string `json:"answering_host"`
16 Count int `json:"count"`
17 Rcode string `json:"rcode"`
18 Rdata string `json:"rdata"`
19 Rrtype string `json:"rrtype"`
20 Type string `json:"type"`
21 }
22
23 type inputJSONstruct struct {
24 DNS map[string]struct {
25 Rdata []rdata `json:"rdata"`
26 } `json:"dns"`
27 TimestampEnd time.Time `json:"timestamp_end"`
28 TimestampStart time.Time `json:"timestamp_start"`
29 }
30
31 // MakeFeverAggregateInputObservations is a MakeObservationFunc that accepts
32 // input in suristasher/FEVER's JSON format.
33 func MakeFeverAggregateInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
34 var in inputJSONstruct
35 var i int64
36 err := json.Unmarshal(inputJSON, &in)
37 if err != nil {
38 log.Warn(err)
39 return nil
40 }
41 for k, v := range in.DNS {
42 select {
43 case <-stop:
44 return nil
45 default:
46 for _, v2 := range v.Rdata {
47 select {
48 case <-stop:
49 return nil
50 default:
51 o := observation.InputObservation{
52 Count: v2.Count,
53 Rdata: v2.Rdata,
54 Rrname: k,
55 Rrtype: v2.Rrtype,
56 SensorID: sensorID,
57 TimestampEnd: in.TimestampEnd,
58 TimestampStart: in.TimestampStart,
59 }
60 i++
61 out <- o
62 }
63 }
64 }
65 }
66 log.Infof("enqueued %d observations", i)
67 return nil
68 }
+114
-0
format/format_fever_test.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 "github.com/sirupsen/logrus/hooks/test"
11 )
12
13 func TestFeverFormatFail(t *testing.T) {
14 hook := test.NewGlobal()
15
16 resultObs := make([]observation.InputObservation, 0)
17 resChan := make(chan observation.InputObservation)
18 go func() {
19 for o := range resChan {
20 resultObs = append(resultObs, o)
21 }
22 }()
23
24 stopChan := make(chan bool)
25 err := MakeFeverAggregateInputObservations([]byte(`babanana`), "foo", resChan, stopChan)
26 if err != nil {
27 t.Fatal(err)
28 }
29 close(resChan)
30 close(stopChan)
31
32 if len(resultObs) != 0 {
33 t.Fail()
34 }
35 if len(hook.Entries) != 1 {
36 t.Fail()
37 }
38 }
39
40 func TestFeverFormatEmpty(t *testing.T) {
41 hook := test.NewGlobal()
42
43 resultObs := make([]observation.InputObservation, 0)
44 resChan := make(chan observation.InputObservation)
45 go func() {
46 for o := range resChan {
47 resultObs = append(resultObs, o)
48 }
49 }()
50
51 stopChan := make(chan bool)
52 err := MakeFeverAggregateInputObservations([]byte(""), "foo", resChan, stopChan)
53 if err != nil {
54 t.Fatal(err)
55 }
56 close(resChan)
57 close(stopChan)
58
59 if len(resultObs) != 0 {
60 t.Fail()
61 }
62 if len(hook.Entries) != 1 {
63 t.Fail()
64 }
65 }
66
67 const exampleInFever = `{
68 "dns": {
69 "foo.bar": {
70 "rdata": [
71 {
72 "rdata": "1.2.3.4",
73 "count":2,
74 "rrtype": "A",
75 "type":"answer"
76 },
77 {
78 "rdata": "1.2.3.5",
79 "count":1,
80 "rrtype": "A",
81 "type":"answer"
82 }
83 ]
84 }
85 },
86 "timestamp_start":"2018-10-26T21:02:20+00:00",
87 "timestamp_end":"2018-10-26T21:03:20+00:00"
88 }`
89
90 func TestFeverFormat(t *testing.T) {
91 hook := test.NewGlobal()
92
93 resultObs := make([]observation.InputObservation, 0)
94 resChan := make(chan observation.InputObservation)
95 go func() {
96 for o := range resChan {
97 resultObs = append(resultObs, o)
98 }
99 }()
100
101 stopChan := make(chan bool)
102 err := MakeFeverAggregateInputObservations([]byte(exampleInFever), "foo", resChan, stopChan)
103 if err != nil {
104 t.Fatal(err)
105 }
106 time.Sleep(500 * time.Millisecond)
107 close(resChan)
108 close(stopChan)
109
110 if len(hook.Entries) != 1 {
111 t.Fail()
112 }
113 }
+72
-0
format/format_gamelinux.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "bufio"
7 "strconv"
8 "strings"
9 "time"
10
11 "github.com/DCSO/balboa/observation"
12
13 log "github.com/sirupsen/logrus"
14 )
15
16 // MakeFjellskaalInputObservations is a MakeObservationFunc that consumes
17 // input in the format as used by https://github.com/gamelinux/passivedns.
18 func MakeFjellskaalInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
19 var i int
20 scanner := bufio.NewScanner(strings.NewReader(string(inputJSON)))
21 for scanner.Scan() {
22 select {
23 case <-stop:
24 return nil
25 default:
26 vals := strings.Split(scanner.Text(), "||")
27 if len(vals) == 9 {
28 times := strings.Split(vals[0], ".")
29 if len(times) != 2 {
30 log.Warn("timestamp does not have form X.X")
31 continue
32 }
33 epoch, err := strconv.Atoi(times[0])
34 if err != nil {
35 log.Warn(err)
36 continue
37 }
38 nsec, err := strconv.Atoi(times[1])
39 if err != nil {
40 log.Warn(err)
41 continue
42 }
43 timestamp := time.Unix(int64(epoch), int64(nsec))
44 rrname := vals[4]
45 rrtype := vals[5]
46 rdata := vals[6]
47 count, err := strconv.Atoi(vals[8])
48 if err != nil {
49 log.Warn(err)
50 continue
51 }
52 o := observation.InputObservation{
53 Count: count,
54 Rdata: strings.TrimRight(rdata, "."),
55 Rrname: strings.TrimRight(rrname, "."),
56 Rrtype: rrtype,
57 SensorID: sensorID,
58 TimestampEnd: timestamp,
59 TimestampStart: timestamp,
60 }
61 i++
62 out <- o
63 } else {
64 log.Warn("number of columns != 9")
65 }
66 }
67 }
68
69 log.Debugf("enqueued %d observations", i)
70 return nil
71 }
+119
-0
format/format_gamelinux_test.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 "github.com/sirupsen/logrus/hooks/test"
11 )
12
13 func TestFjellskaalFormatFail(t *testing.T) {
14 hook := test.NewGlobal()
15
16 resultObs := make([]observation.InputObservation, 0)
17 resChan := make(chan observation.InputObservation)
18 go func() {
19 for o := range resChan {
20 resultObs = append(resultObs, o)
21 }
22 }()
23
24 stopChan := make(chan bool)
25 err := MakeFjellskaalInputObservations([]byte("1322849924||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||5"), "foo", resChan, stopChan)
26 if err != nil {
27 t.Fatal(err)
28 }
29 err = MakeFjellskaalInputObservations([]byte("1322849924||10.1.1.1||8.8.8.8||upload.youtube.com.||A||74.125.43.117||46587||5"), "foo", resChan, stopChan)
30 if err != nil {
31 t.Fatal(err)
32 }
33 err = MakeFjellskaalInputObservations([]byte("X.2332||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||5"), "foo", resChan, stopChan)
34 if err != nil {
35 t.Fatal(err)
36 }
37 err = MakeFjellskaalInputObservations([]byte("X.X||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||5"), "foo", resChan, stopChan)
38 if err != nil {
39 t.Fatal(err)
40 }
41 err = MakeFjellskaalInputObservations([]byte("23232.X||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||5"), "foo", resChan, stopChan)
42 if err != nil {
43 t.Fatal(err)
44 }
45 err = MakeFjellskaalInputObservations([]byte("1322849924.244555||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||bar"), "foo", resChan, stopChan)
46 if err != nil {
47 t.Fatal(err)
48 }
49 close(resChan)
50 close(stopChan)
51
52 if len(resultObs) != 0 {
53 t.Fail()
54 }
55 if len(hook.Entries) != 6 {
56 t.Fail()
57 }
58 }
59
60 func TestFjellskaalFormatEmpty(t *testing.T) {
61 hook := test.NewGlobal()
62
63 resultObs := make([]observation.InputObservation, 0)
64 resChan := make(chan observation.InputObservation)
65 go func() {
66 for o := range resChan {
67 resultObs = append(resultObs, o)
68 }
69 }()
70
71 stopChan := make(chan bool)
72 err := MakeFjellskaalInputObservations([]byte(""), "foo", resChan, stopChan)
73 if err != nil {
74 t.Fatal(err)
75 }
76 close(resChan)
77 close(stopChan)
78
79 if len(resultObs) != 0 {
80 t.Fail()
81 }
82 if len(hook.Entries) != 0 {
83 t.Fail()
84 }
85 }
86
87 const exampleInFjellskaal = `1322849924.408856||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.117||46587||5
88 1322849924.408857||10.1.1.1||8.8.8.8||IN||upload.youtube.com.||A||74.125.43.116||420509||5
89 1322849924.408858||10.1.1.1||8.8.8.8||IN||www.adobe.com.||CNAME||www.wip4.adobe.com.||43200||8
90 1322849924.408859||10.1.1.1||8.8.8.8||IN||www.adobe.com.||A||193.104.215.61||43200||8
91 1322849924.408860||10.1.1.1||8.8.8.8||IN||i1.ytimg.com.||CNAME||ytimg.l.google.com.||43200||3
92 1322849924.408861||10.1.1.1||8.8.8.8||IN||clients1.google.com.||A||173.194.32.3||43200||2
93 `
94
95 func TestFjellskaalFormat(t *testing.T) {
96 hook := test.NewGlobal()
97
98 resultObs := make([]observation.InputObservation, 0)
99 resChan := make(chan observation.InputObservation)
100 go func() {
101 for o := range resChan {
102 resultObs = append(resultObs, o)
103 }
104 }()
105
106 stopChan := make(chan bool)
107 err := MakeFjellskaalInputObservations([]byte(exampleInFjellskaal), "foo", resChan, stopChan)
108 if err != nil {
109 t.Fatal(err)
110 }
111 time.Sleep(500 * time.Millisecond)
112 close(resChan)
113 close(stopChan)
114
115 if len(hook.Entries) != 0 {
116 t.Fail()
117 }
118 }
+64
-0
format/format_gopassivedns.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "encoding/json"
7 "net"
8 "time"
9
10 "github.com/DCSO/balboa/observation"
11
12 log "github.com/sirupsen/logrus"
13 )
14
15 type dnsLogEntry struct {
16 QueryID uint16 `json:"query_id"`
17 ResponseCode int `json:"rcode"`
18 Question string `json:"q"`
19 QuestionType string `json:"qtype"`
20 Answer string `json:"a"`
21 AnswerType string `json:"atype"`
22 TTL uint32 `json:"ttl"`
23 Server net.IP `json:"dst"`
24 Client net.IP `json:"src"`
25 Timestamp string `json:"tstamp"`
26 Elapsed int64 `json:"elapsed"`
27 ClientPort string `json:"sport"`
28 Level string `json:"level"`
29 Length int `json:"bytes"`
30 Proto string `json:"protocol"`
31 Truncated bool `json:"truncated"`
32 AuthoritativeAnswer bool `json:"aa"`
33 RecursionDesired bool `json:"rd"`
34 RecursionAvailable bool `json:"ra"`
35 }
36
37 // MakeGopassivednsInputObservations is a MakeObservationFunc that accepts
38 // input in the format as generated by https://github.com/Phillipmartin/gopassivedns.
39 func MakeGopassivednsInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
40 var in dnsLogEntry
41 err := json.Unmarshal(inputJSON, &in)
42 if err != nil {
43 log.Warn(err)
44 return nil
45 }
46 tst, err := time.Parse("2006-01-02 15:04:05.999999 -0700 MST", in.Timestamp)
47 if err != nil {
48 log.Info(err)
49 return nil
50 }
51 o := observation.InputObservation{
52 Count: 1,
53 Rdata: in.Answer,
54 Rrname: in.Question,
55 Rrtype: in.AnswerType,
56 SensorID: sensorID,
57 TimestampEnd: tst,
58 TimestampStart: tst,
59 }
60 out <- o
61 log.Debug("enqueued 1 observation")
62 return nil
63 }
+94
-0
format/format_gopassivedns_test.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 "github.com/sirupsen/logrus/hooks/test"
11 )
12
13 func TestGopassivednsFormatFail(t *testing.T) {
14 hook := test.NewGlobal()
15
16 resultObs := make([]observation.InputObservation, 0)
17 resChan := make(chan observation.InputObservation)
18 go func() {
19 for o := range resChan {
20 resultObs = append(resultObs, o)
21 }
22 }()
23
24 stopChan := make(chan bool)
25 err := MakeGopassivednsInputObservations([]byte(`{"query_id":43264,"rcode":0,"q":"github.com","qtype":"A","a":"192.30.253.112","atype":"A","ttl":60,"dst":"9.9.9.9","src":"192.168.1.79","tstamp":"2018-10-26 19 +0000 UTC","elapsed":35879000,"sport":"40651","level":"","bytes":102,"protocol":"udp","truncated":false,"aa":false,"rd":true,"ra":false}`), "foo", resChan, stopChan)
26 if err != nil {
27 t.Fatal(err)
28 }
29 close(resChan)
30 close(stopChan)
31
32 if len(resultObs) != 0 {
33 t.Fail()
34 }
35 if len(hook.Entries) != 1 {
36 t.Fail()
37 }
38 }
39
40 func TestGopassivednsFormatEmpty(t *testing.T) {
41 hook := test.NewGlobal()
42
43 resultObs := make([]observation.InputObservation, 0)
44 resChan := make(chan observation.InputObservation)
45 go func() {
46 for o := range resChan {
47 resultObs = append(resultObs, o)
48 }
49 }()
50
51 stopChan := make(chan bool)
52 err := MakeGopassivednsInputObservations([]byte(""), "foo", resChan, stopChan)
53 if err != nil {
54 t.Fatal(err)
55 }
56 close(resChan)
57 close(stopChan)
58
59 if len(resultObs) != 0 {
60 t.Fail()
61 }
62 if len(hook.Entries) != 1 {
63 t.Fail()
64 }
65 }
66
67 const exampleInGopassivedns = `{"query_id":43264,"rcode":0,"q":"github.com","qtype":"A","a":"192.30.253.112","atype":"A","ttl":60,"dst":"9.9.9.9","src":"192.168.1.79","tstamp":"2018-10-26 19:32:36.141184 +0000 UTC","elapsed":35879000,"sport":"40651","level":"","bytes":102,"protocol":"udp","truncated":false,"aa":false,"rd":true,"ra":false}
68 `
69
70 func TestGopassivednsFormat(t *testing.T) {
71 hook := test.NewGlobal()
72
73 resultObs := make([]observation.InputObservation, 0)
74 resChan := make(chan observation.InputObservation)
75 go func() {
76 for o := range resChan {
77 resultObs = append(resultObs, o)
78 }
79 }()
80
81 stopChan := make(chan bool)
82 err := MakeGopassivednsInputObservations([]byte(exampleInGopassivedns), "foo", resChan, stopChan)
83 if err != nil {
84 t.Fatal(err)
85 }
86 time.Sleep(500 * time.Millisecond)
87 close(resChan)
88 close(stopChan)
89
90 if len(hook.Entries) != 0 {
91 t.Fail()
92 }
93 }
+69
-0
format/format_packetbeat.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "encoding/json"
7 "strings"
8 "time"
9
10 "github.com/DCSO/balboa/observation"
11
12 log "github.com/sirupsen/logrus"
13 )
14
15 type pbEntry struct {
16 Type string `json:"type"`
17 Timestamp string `json:"@timestamp"`
18 DNS struct {
19 Answers []struct {
20 Name string `json:"name"`
21 Class string `json:"class"`
22 Type string `json:"type"`
23 Data string `json:"data"`
24 TTL string `json:"ttl"`
25 } `json:"answers"`
26 } `json:"dns"`
27 }
28
29 // MakePacketbeatInputObservations is a MakeObservationFunc that accepts a
30 // JSON format from Packetbeat via Logstash. See doc/packetbeat_config.txt
31 // for more information.
32 func MakePacketbeatInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
33 var in pbEntry
34 var i int
35 err := json.Unmarshal(inputJSON, &in)
36 if err != nil {
37 log.Warn(err)
38 return nil
39 }
40 if in.Type != "dns" {
41 return nil
42 }
43 tst, err := time.Parse("2006-01-02T15:04:05.999Z07", in.Timestamp)
44 if err != nil {
45 log.Warn(err)
46 return nil
47 }
48 for _, answer := range in.DNS.Answers {
49 select {
50 case <-stop:
51 return nil
52 default:
53 o := observation.InputObservation{
54 Count: 1,
55 Rdata: strings.TrimRight(answer.Data, "."),
56 Rrname: strings.TrimRight(answer.Name, "."),
57 Rrtype: answer.Type,
58 SensorID: sensorID,
59 TimestampEnd: tst,
60 TimestampStart: tst,
61 }
62 i++
63 out <- o
64 }
65 }
66 log.Infof("enqueued %d observations", i)
67 return nil
68 }
+144
-0
format/format_packetbeat_test.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 "github.com/sirupsen/logrus/hooks/test"
11 )
12
13 func TestPacketbeatFormatFail(t *testing.T) {
14 hook := test.NewGlobal()
15
16 resultObs := make([]observation.InputObservation, 0)
17 resChan := make(chan observation.InputObservation)
18 go func() {
19 for o := range resChan {
20 resultObs = append(resultObs, o)
21 }
22 }()
23
24 stopChan := make(chan bool)
25 err := MakePacketbeatInputObservations([]byte(`babanana`), "foo", resChan, stopChan)
26 if err != nil {
27 t.Fatal(err)
28 }
29 err = MakePacketbeatInputObservations([]byte(exampleInPacketbeatInvalidTimestamp), "foo", resChan, stopChan)
30 if err != nil {
31 t.Fatal(err)
32 }
33 err = MakePacketbeatInputObservations([]byte(exampleInPacketbeatInvalidType), "foo", resChan, stopChan)
34 if err != nil {
35 t.Fatal(err)
36 }
37 close(resChan)
38 close(stopChan)
39
40 if len(resultObs) != 0 {
41 t.Fail()
42 }
43 if len(hook.Entries) != 2 {
44 t.Fail()
45 }
46 }
47
48 func TestPacketbeatFormatEmpty(t *testing.T) {
49 hook := test.NewGlobal()
50
51 resultObs := make([]observation.InputObservation, 0)
52 resChan := make(chan observation.InputObservation)
53 go func() {
54 for o := range resChan {
55 resultObs = append(resultObs, o)
56 }
57 }()
58
59 stopChan := make(chan bool)
60 err := MakePacketbeatInputObservations([]byte(""), "foo", resChan, stopChan)
61 if err != nil {
62 t.Fatal(err)
63 }
64 close(resChan)
65 close(stopChan)
66
67 if len(resultObs) != 0 {
68 t.Fail()
69 }
70 if len(hook.Entries) != 1 {
71 t.Fail()
72 }
73 }
74
75 const exampleInPacketbeatInvalidTimestamp = `{
76 "type": "dns",
77 "dns": {
78 "answers": [{
79 "name": "foo.bar.",
80 "data": "1.2.3.4.",
81 "type": "A",
82 "class":"foo"
83 }]
84
85 },
86 "@timestamp": "2018-10-26T2"
87 }`
88
89 const exampleInPacketbeatInvalidType = `{
90 "type": "whatever",
91 "dns": {
92 "answers": [{
93 "name": "foo.bar.",
94 "data": "1.2.3.4.",
95 "type": "A",
96 "class":"foo"
97 }]
98
99 },
100 "@timestamp": "2018-10-26T21:03:20.222Z"
101 }`
102
103 const exampleInPacketbeat = `{
104 "type": "dns",
105 "dns": {
106 "answers": [{
107 "name": "foo.bar.",
108 "data": "1.2.3.4.",
109 "type": "A",
110 "class":"foo"
111 }]
112
113 },
114 "@timestamp": "2018-10-26T21:03:20.222Z"
115 }`
116
117 func TestPacketbeatFormat(t *testing.T) {
118 hook := test.NewGlobal()
119
120 resultObs := make([]observation.InputObservation, 0)
121 resChan := make(chan observation.InputObservation, 100)
122
123 stopChan := make(chan bool)
124 err := MakePacketbeatInputObservations([]byte(exampleInPacketbeat), "foo", resChan, stopChan)
125 if err != nil {
126 t.Fatal(err)
127 }
128 time.Sleep(500 * time.Millisecond)
129 close(resChan)
130 close(stopChan)
131
132 for o := range resChan {
133 resultObs = append(resultObs, o)
134 }
135
136 if len(resultObs) != 1 {
137 t.Fail()
138 }
139
140 if len(hook.Entries) != 1 {
141 t.Fail()
142 }
143 }
+108
-0
format/format_suricata.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "encoding/json"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10
11 log "github.com/sirupsen/logrus"
12 )
13
14 type suriDNSEntry struct {
15 EventType string `json:"event_type"`
16 Timestamp string `json:"timestamp"`
17 DNS struct {
18 Type string `json:"type"`
19 Version int `json:"version"`
20 Rrtype string `json:"rrtype"`
21 Rcode string `json:"rcode"`
22 Rrname string `json:"rrname"`
23 TTL int `json:"ttl"`
24 Rdata string `json:"rdata"`
25 Answers []struct {
26 Rrname string `json:"rrname"`
27 Rrtype string `json:"rrtype"`
28 TTL int `json:"ttl"`
29 Rdata string `json:"rdata"`
30 } `json:"answers"`
31 Grouped map[string]([]string) `json:"grouped"`
32 } `json:"dns"`
33 }
34
35 // MakeSuricataInputObservations is a MakeObservationFunc that accepts input
36 // in Suricata's EVE JSON format (DNS type version 1 and 2 are supported).
37 func MakeSuricataInputObservations(inputJSON []byte, sensorID string, out chan observation.InputObservation, stop chan bool) error {
38 var in suriDNSEntry
39 var i int
40 err := json.Unmarshal(inputJSON, &in)
41 if err != nil {
42 log.Warn(err)
43 return nil
44 }
45 if in.EventType != "dns" {
46 return nil
47 }
48 if in.DNS.Type != "answer" {
49 return nil
50 }
51 tst, err := time.Parse("2006-01-02T15:04:05.999999-0700", in.Timestamp)
52 if err != nil {
53 log.Warn(err)
54 return nil
55 }
56 if in.DNS.Version == 2 {
57 // v2 format
58 if len(in.DNS.Answers) > 0 {
59 // "detailed" format
60 for _, answer := range in.DNS.Answers {
61 o := observation.InputObservation{
62 Count: 1,
63 Rdata: answer.Rdata,
64 Rrname: answer.Rrname,
65 Rrtype: answer.Rrtype,
66 SensorID: sensorID,
67 TimestampEnd: tst,
68 TimestampStart: tst,
69 }
70 i++
71 out <- o
72 }
73 } else {
74 // "grouped" format
75 for rrtype, rdataArr := range in.DNS.Grouped {
76 for _, rdata := range rdataArr {
77 o := observation.InputObservation{
78 Count: 1,
79 Rdata: rdata,
80 Rrname: in.DNS.Rrname,
81 Rrtype: rrtype,
82 SensorID: sensorID,
83 TimestampEnd: tst,
84 TimestampStart: tst,
85 }
86 i++
87 out <- o
88 }
89 }
90 }
91 } else {
92 // v1 format
93 o := observation.InputObservation{
94 Count: 1,
95 Rdata: in.DNS.Rdata,
96 Rrname: in.DNS.Rrname,
97 Rrtype: in.DNS.Rrtype,
98 SensorID: sensorID,
99 TimestampEnd: tst,
100 TimestampStart: tst,
101 }
102 i++
103 out <- o
104 }
105 log.Infof("enqueued %d observations", i)
106 return nil
107 }
+274
-0
format/format_suricata_test.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package format
4
5 import (
6 "testing"
7 "time"
8
9 "github.com/DCSO/balboa/observation"
10 "github.com/sirupsen/logrus/hooks/test"
11 )
12
13 func TestSuricataFormatFail(t *testing.T) {
14 hook := test.NewGlobal()
15
16 resultObs := make([]observation.InputObservation, 0)
17 resChan := make(chan observation.InputObservation)
18 go func() {
19 for o := range resChan {
20 resultObs = append(resultObs, o)
21 }
22 }()
23
24 stopChan := make(chan bool)
25 err := MakeSuricataInputObservations([]byte(`babanana`), "foo", resChan, stopChan)
26 if err != nil {
27 t.Fatal(err)
28 }
29 err = MakeSuricataInputObservations([]byte(exampleInSuricataInvalidTimestamp), "foo", resChan, stopChan)
30 if err != nil {
31 t.Fatal(err)
32 }
33 err = MakeSuricataInputObservations([]byte(exampleInSuricataInvalidType), "foo", resChan, stopChan)
34 if err != nil {
35 t.Fatal(err)
36 }
37 err = MakeSuricataInputObservations([]byte(exampleInSuricataInvalidType2), "foo", resChan, stopChan)
38 if err != nil {
39 t.Fatal(err)
40 }
41 close(resChan)
42 close(stopChan)
43
44 if len(resultObs) != 0 {
45 t.Fail()
46 }
47 if len(hook.Entries) != 2 {
48 t.Fail()
49 }
50 }
51
52 func TestSuricataFormatEmpty(t *testing.T) {
53 hook := test.NewGlobal()
54
55 resultObs := make([]observation.InputObservation, 0)
56 resChan := make(chan observation.InputObservation)
57 go func() {
58 for o := range resChan {
59 resultObs = append(resultObs, o)
60 }
61 }()
62
63 stopChan := make(chan bool)
64 err := MakeSuricataInputObservations([]byte(""), "foo", resChan, stopChan)
65 if err != nil {
66 t.Fatal(err)
67 }
68 close(resChan)
69 close(stopChan)
70
71 if len(resultObs) != 0 {
72 t.Fail()
73 }
74 if len(hook.Entries) != 1 {
75 t.Fail()
76 }
77 }
78
79 const exampleInSuricataInvalidTimestamp = `{
80 "timestamp": "2009-11-24T21:",
81 "event_type": "dns",
82 "src_ip": "192.168.2.7",
83 "src_port": 53,
84 "dest_ip": "x.x.250.50",
85 "dest_port": 23242,
86 "proto": "UDP",
87 "dns": {
88 "type": "answer",
89 "id":16000,
90 "flags":"8180",
91 "qr":true,
92 "rd":true,
93 "ra":true,
94 "rcode":"NOERROR",
95 "rrname": "twitter.com",
96 "rrtype":"A",
97 "ttl":8,
98 "rdata": "199.16.156.6"
99 }
100 }`
101
102 const exampleInSuricataInvalidType = `{
103 "timestamp": "2009-11-24T21:27:09.534255-0100",
104 "event_type": "foo",
105 "src_ip": "192.168.2.7",
106 "src_port": 53,
107 "dest_ip": "x.x.250.50",
108 "dest_port": 23242,
109 "proto": "UDP",
110 "dns": {
111 "type": "answer",
112 "id":16000,
113 "flags":"8180",
114 "qr":true,
115 "rd":true,
116 "ra":true,
117 "rcode":"NOERROR",
118 "rrname": "twitter.com",
119 "rrtype":"A",
120 "ttl":8,
121 "rdata": "199.16.156.6"
122 }
123 }`
124
125 const exampleInSuricataInvalidType2 = `{
126 "timestamp": "2009-11-24T21:27:09.534255-0100",
127 "event_type": "dns",
128 "src_ip": "192.168.2.7",
129 "src_port": 53,
130 "dest_ip": "x.x.250.50",
131 "dest_port": 23242,
132 "proto": "UDP",
133 "dns": {
134 "type": "foo",
135 "id":16000,
136 "flags":"8180",
137 "qr":true,
138 "rd":true,
139 "ra":true,
140 "rcode":"NOERROR",
141 "rrname": "twitter.com",
142 "rrtype":"A",
143 "ttl":8,
144 "rdata": "199.16.156.6"
145 }
146 }`
147
148 const exampleInSuricataV2 = `{
149 "timestamp": "2009-11-24T21:27:09.534255-0100",
150 "event_type": "dns",
151 "src_ip": "192.168.2.7",
152 "src_port": 53,
153 "dest_ip": "x.x.250.50",
154 "dest_port": 23242,
155 "proto": "UDP",
156 "dns": {
157 "version": 2,
158 "type": "answer",
159 "id": 45444,
160 "flags": "8180",
161 "qr": true,
162 "rd": true,
163 "ra": true,
164 "rcode": "NOERROR",
165 "answers": [
166 {
167 "rrname": "www.suricata-ids.org",
168 "rrtype": "CNAME",
169 "ttl": 3324,
170 "rdata": "suricata-ids.org"
171 },
172 {
173 "rrname": "suricata-ids.org",
174 "rrtype": "A",
175 "ttl": 10,
176 "rdata": "192.0.78.24"
177 },
178 {
179 "rrname": "suricata-ids.org",
180 "rrtype": "A",
181 "ttl": 10,
182 "rdata": "192.0.78.25"
183 }
184 ]
185 }
186 }`
187
188 const exampleInSuricataV2Grouped = `{
189 "timestamp": "2009-11-24T21:27:09.534255-0100",
190 "event_type": "dns",
191 "src_ip": "192.168.2.7",
192 "src_port": 53,
193 "dest_ip": "x.x.250.50",
194 "dest_port": 23242,
195 "proto": "UDP",
196 "dns": {
197 "version": 2,
198 "type": "answer",
199 "id": 18523,
200 "flags": "8180",
201 "qr": true,
202 "rd": true,
203 "ra": true,
204 "rcode": "NOERROR",
205 "grouped": {
206 "A": [
207 "192.0.78.24",
208 "192.0.78.25"
209 ],
210 "CNAME": [
211 "suricata-ids.org"
212 ]
213 }
214 }
215 }`
216
217 const exampleInSuricataV1 = `{
218 "timestamp": "2009-11-24T21:27:09.534255-0100",
219 "event_type": "dns",
220 "src_ip": "192.168.2.7",
221 "src_port": 53,
222 "dest_ip": "x.x.250.50",
223 "dest_port": 23242,
224 "proto": "UDP",
225 "dns": {
226 "type": "answer",
227 "id":16000,
228 "flags":"8180",
229 "qr":true,
230 "rd":true,
231 "ra":true,
232 "rcode":"NOERROR",
233 "rrname": "twitter.com",
234 "rrtype":"A",
235 "ttl":8,
236 "rdata": "199.16.156.6"
237 }
238 }`
239
240 func TestSuricataFormat(t *testing.T) {
241 hook := test.NewGlobal()
242
243 resultObs := make([]observation.InputObservation, 0)
244 resChan := make(chan observation.InputObservation, 100)
245
246 stopChan := make(chan bool)
247 err := MakeSuricataInputObservations([]byte(exampleInSuricataV1), "foo", resChan, stopChan)
248 if err != nil {
249 t.Fatal(err)
250 }
251 err = MakeSuricataInputObservations([]byte(exampleInSuricataV2), "foo", resChan, stopChan)
252 if err != nil {
253 t.Fatal(err)
254 }
255 err = MakeSuricataInputObservations([]byte(exampleInSuricataV2Grouped), "foo", resChan, stopChan)
256 if err != nil {
257 t.Fatal(err)
258 }
259 time.Sleep(500 * time.Millisecond)
260 close(resChan)
261 close(stopChan)
262
263 for o := range resChan {
264 resultObs = append(resultObs, o)
265 }
266
267 if len(resultObs) != 7 {
268 t.Fail()
269 }
270 if len(hook.Entries) != 3 {
271 t.Fail()
272 }
273 }
+29
-0
observation/input_observation.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package observation
4
5 import (
6 "time"
7 )
8
9 // InputObservation is a minimal, small observation structure to be used as
10 // the minimal common input type for all consumers.
11 type InputObservation struct {
12 Count int
13 Rcode string
14 Rdata string
15 Rrtype string
16 Rrname string
17 SensorID string
18 TimestampEnd time.Time
19 TimestampStart time.Time
20 }
21
22 // InChan is the global input channel delivering InputObservations from
23 // feeders to consumers.
24 var InChan chan InputObservation
25
26 func init() {
27 InChan = make(chan InputObservation, 50000)
28 }
+21
-0
observation/observation.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package observation
4
5 import (
6 uuid "github.com/satori/go.uuid"
7 )
8
9 // Observation represents a DNS answer, potentially repeated, observed on a
10 // given sensor stating a specific RR set.
11 type Observation struct {
12 ID uuid.UUID `json:"-"`
13 Count int `json:"count"`
14 FirstSeen int `json:"time_first"`
15 LastSeen int `json:"time_last"`
16 RRType string `json:"rrtype"`
17 RRName string `json:"rrname"`
18 RData string `json:"rdata"`
19 SensorID string `json:"sensor_id"`
20 }
+423
-0
query/query_graphql.go less more
0 // balboa
1 // Copyright (c) 2018, DCSO GmbH
2
3 package query
4
5 import (
6 "context"
7 "fmt"
8 "net/http"
9 "runtime"
10 "time"
11
12 "github.com/DCSO/balboa/db"
13 "github.com/DCSO/balboa/observation"
14
15 graphql "github.com/graph-gophers/graphql-go"
16 "github.com/graph-gophers/graphql-go/errors"
17 "github.com/graph-gophers/graphql-go/relay"
18 uuid "github.com/satori/go.uuid"
19 log "github.com/sirupsen/logrus"
20 )
21
22 const (
23 txtSchema = `# A DNS resource record type.
24 enum RRType {
25 A
26 A6
27 AAAA
28 AFSDB
29 ALIAS
30 APL
31 AXFR
32 CAA
33 CDNSKEY
34 CDS
35 CERT
36 CNAME
37 DHCID
38 DLV
39 DNAME
40 DNSKEY
41 DS
42 HINFO
43 HIP
44 IPSECKEY
45 IXFR
46 KEY
47 KX
48 LOC
49 MX
50 NAPTR
51 NS
52 NSEC
53 NSEC3
54 NSEC3PARAM
55 OPENPGPKEY
56 OPT
57 PTR
58 RRSIG
59 RP
60 SIG
61 SOA
62 SPF
63 SRV
64 SSHFP
65 TA
66 TKEY
67 TLSA
68 TSIG
69 TXT
70 URI
71 }
72
73 # A single observation, unique for the combination of sensor, rrname,
74 # rdata and rrtype. Corresponds, roughly, to a pDNS COF item, but with
75 # additional Aliases (linked via IP in A/AAAA records).
76 type Entry {
77 # The number of observed occurrences of this observation.
78 count: Int!
79
80 # The RRName seen in this observation.
81 rrname: String!
82
83 # The RRType seen in this observation.
84 rrtype: RRType
85
86 # The RData seen in this observation.
87 rdata: String!
88
89 # Time this observation was first seen, as Unix timestamp.
90 time_first: Int!
91
92 # Time this observation was first seen, as RFC 3339 formatted time.
93 time_first_rfc3339: String!
94
95 # Time this observation was last seen, as Unix timestamp.
96 time_last: Int!
97
98 # Time this observation was last seen, as RFC 3339 formatted time.
99 time_last_rfc3339: String!
100
101 # Some identifier describing the source of this observation.
102 sensor_id: String
103
104 # Entries referencing the same IP (for A/AAAA) observed on the same
105 # sensor.
106 aliases: [LeafEntry]
107 }
108
109 # A single observation, unique for the combination of sensor, rrname,
110 # rdata and rrtype. Corresponds, roughly, to a pDNS COF item.
111 type LeafEntry {
112 # The number of observed occurrences of this observation.
113 count: Int!
114
115 # The RRName seen in this observation.
116 rrname: String!
117
118 # The RRType seen in this observation.
119 rrtype: RRType
120
121 # The RData seen in this observation.
122 rdata: String!
123
124 # Time this observation was first seen, as Unix timestamp.
125 time_first: Int!
126
127 # Time this observation was first seen, as RFC 3339 formatted time.
128 time_first_rfc3339: String!
129
130 # Time this observation was last seen, as Unix timestamp.
131 time_last: Int!
132
133 # Time this observation was last seen, as RFC 3339 formatted time.
134 time_last_rfc3339: String!
135
136 # Some identifier describing the source of this observation.
137 sensor_id: String
138 }
139
140 input EntryInput {
141 # The number of observed occurrences of this observation.
142 count: Int!
143
144 # The RRName seen in this observation.
145 rrname: String!
146
147 # The RRType seen in this observation.
148 rrtype: RRType!
149
150 # The RData seen in this observation.
151 rdata: String!
152
153 # Time this observation was first seen, as Unix timestamp.
154 time_first: Int!
155
156 # Time this observation was last seen, as Unix timestamp.
157 time_last: Int!
158
159 # Some identifier describing the source of this observation.
160 sensor_id: String!
161 }
162
163 # Some runtime values describing the current state of the database.
164 type Stats {
165 # Total number of keys in the database.
166 total_count: Int!
167
168 # Number of concurrent goroutines in the server instance.
169 num_goroutines: Int!
170 }
171
172 type Query {
173 # Returns a set of observations satisfying the given query parameters.
174 # Providing rdata, rrname, rrtype and/or sensor_id will restrict the
175 # results to the set of observations that match all of the given
176 # constraints.
177 entries(rdata: String, rrname: String, rrtype: RRType, sensor_id: String): [Entry]
178
179 # Returns some runtime values describing the current state of the database.
180 stats(): Stats
181 }
182
183 type Mutation {
184 announceObservation(observation: EntryInput!): Entry!
185 }
186
187 schema {
188 query: Query
189 mutation: Mutation
190 }`
191 )
192
193 // GraphQLFrontend represents a concurrent component that provides a GraphQL
194 // query interface for the database.
195 type GraphQLFrontend struct {
196 Server *http.Server
197 IsRunning bool
198 }
199
200 // Resolver is just used to bundle top level methods.
201 type Resolver struct{}
202
203 type entryInputArgs struct {
204 Count int
205 FirstSeen int
206 LastSeen int
207 RRType string
208 RRName string
209 RData string
210 SensorID string
211 }
212
213 // Entries returns a collection of Entry resolvers, given parameters such as
214 // Rdata, RRname, RRtype and sensor ID.
215 func (r *Resolver) Entries(args struct {
216 Rdata *string
217 Rrname *string
218 Rrtype *string
219 SensorID *string
220 }) (*[]*EntryResolver, error) {
221 startTime := time.Now()
222 defer func() {
223 var rdata, rrname, rrtype, sensorID string
224 if args.Rdata != nil {
225 rdata = *args.Rdata
226 } else {
227 rdata = ("nil")
228 }
229 if args.Rrname != nil {
230 rrname = *args.Rrname
231 } else {
232 rrname = ("nil")
233 }
234 if args.Rrtype != nil {
235 rrtype = *args.Rrtype
236 } else {
237 rrtype = ("nil")
238 }
239 if args.SensorID != nil {
240 sensorID = *args.SensorID
241 } else {
242 sensorID = ("nil")
243 }
244 log.Debugf("finished query for (%s/%s/%s/%s) in %v", rdata, rrname, rrtype, sensorID, time.Since(startTime))
245 }()
246
247 l := make([]*EntryResolver, 0)
248 if args.Rdata == nil && args.Rrname == nil {
249 return nil, &errors.QueryError{
250 Message: "at least one of the 'rdata' or 'rrname' parameters is required",
251 }
252 }
253 results, err := db.ObservationDB.Search(args.Rdata, args.Rrname, args.Rrtype, args.SensorID)
254 if err != nil {
255 return nil, err
256 }
257 for _, r := range results {
258 er := EntryResolver{
259 entry: r,
260 }
261 l = append(l, &er)
262 }
263 return &l, nil
264 }
265
266 // AnnounceObservation is a mutation that adds a single new observation
267 // to the database.
268 func (r *Resolver) AnnounceObservation(args struct {
269 Observation struct {
270 Count int32
271 TimeFirst int32
272 TimeLast int32
273 RRType string
274 RRName string
275 RData string
276 SensorID string
277 }
278 }) *EntryResolver {
279 inObs := observation.InputObservation{
280 Count: int(args.Observation.Count),
281 TimestampStart: time.Unix(int64(args.Observation.TimeFirst), 0),
282 TimestampEnd: time.Unix(int64(args.Observation.TimeLast), 0),
283 Rrname: args.Observation.RRName,
284 Rrtype: args.Observation.RRType,
285 Rdata: args.Observation.RData,
286 SensorID: args.Observation.SensorID,
287 }
288 resObs := db.ObservationDB.AddObservation(inObs)
289 return &EntryResolver{
290 entry: resObs,
291 }
292 }
293
294 // Stats returns a Stats resolver.
295 func (r *Resolver) Stats() (*StatsResolver, error) {
296 return &StatsResolver{}, nil
297 }
298
299 // StatsResolver is a resolver for the Stats type.
300 type StatsResolver struct {
301 totalCount int32
302 }
303
304 // TotalCount returns the total number of keys in the database.
305 func (r *StatsResolver) TotalCount() int32 {
306 val, err := db.ObservationDB.TotalCount()
307 if err != nil {
308 log.Error(err)
309 }
310 return int32(val)
311 }
312
313 // NumGoroutines returns the number of currently running goroutines
314 // in balboa.
315 func (r *StatsResolver) NumGoroutines() int32 {
316 return int32(runtime.NumGoroutine())
317 }
318
319 // EntryResolver is a resolver for the Entry type.
320 type EntryResolver struct {
321 entry observation.Observation
322 }
323
324 // ID returns the ID field of the corresponding entry.
325 func (r *EntryResolver) ID() graphql.ID {
326 id, _ := uuid.NewV4()
327 return graphql.ID(id.String())
328 }
329
330 // RRName returns the RRName field of the corresponding entry.
331 func (r *EntryResolver) RRName() string {
332 return r.entry.RRName
333 }
334
335 // Rdata returns the Rdata field of the corresponding entry.
336 func (r *EntryResolver) Rdata() string {
337 return r.entry.RData
338 }
339
340 // RRType returns the RRType field of the corresponding entry.
341 func (r *EntryResolver) RRType() *string {
342 return &r.entry.RRType
343 }
344
345 // Count returns the Count field of the corresponding entry.
346 func (r *EntryResolver) Count() int32 {
347 return int32(r.entry.Count)
348 }
349
350 // TimeFirst returns the first seen timestamp of the corresponding entry.
351 func (r *EntryResolver) TimeFirst() int32 {
352 return int32(r.entry.FirstSeen)
353 }
354
355 // TimeFirstRFC3339 returns first seen time, as RFC 3339 string, of the corresponding entry.
356 func (r *EntryResolver) TimeFirstRFC3339() string {
357 return time.Unix(int64(r.entry.FirstSeen), 0).Format(time.RFC3339)
358 }
359
360 // TimeLast returns the last seen timestamp of the corresponding entry.
361 func (r *EntryResolver) TimeLast() int32 {
362 return int32(r.entry.LastSeen)
363 }
364
365 // TimeLastRFC3339 returns last seen time, as RFC 3339 string, of the corresponding entry.
366 func (r *EntryResolver) TimeLastRFC3339() string {
367 return time.Unix(int64(r.entry.LastSeen), 0).Format(time.RFC3339)
368 }
369
370 // SensorID returns the sensor ID field of the corresponding entry.
371 func (r *EntryResolver) SensorID() *string {
372 return &r.entry.SensorID
373 }
374
375 // Aliases returns resolvers for Entries with the same IPs in Rdata (for
376 // A/AAAA type entries).
377 func (r *EntryResolver) Aliases() *[]*EntryResolver {
378 l := make([]*EntryResolver, 0)
379 if !(r.entry.RRType == "A" || r.entry.RRType == "AAAA") {
380 return nil
381 }
382 results, err := db.ObservationDB.Search(&r.entry.RData, nil, nil, &r.entry.SensorID)
383 if err != nil {
384 return nil
385 }
386 for _, rs := range results {
387 if rs.RRName != r.entry.RRName {
388 er := EntryResolver{
389 entry: rs,
390 }
391 l = append(l, &er)
392 }
393 }
394 return &l
395 }
396
397 // Run starts this instance of a GraphQLFrontend in the background, accepting
398 // new requests on the configured port.
399 func (g *GraphQLFrontend) Run(port int) {
400 schema := graphql.MustParseSchema(txtSchema, &Resolver{})
401 g.Server = &http.Server{
402 Addr: fmt.Sprintf(":%v", port),
403 Handler: &relay.Handler{Schema: schema},
404 ReadTimeout: 5 * time.Second,
405 WriteTimeout: 10 * time.Second,
406 }
407 log.Infof("serving GraphQL on port %v", port)
408 go func() {
409 err := g.Server.ListenAndServe()
410 if err != nil {
411 log.Info(err)
412 }
413 g.IsRunning = true
414 }()
415 }
416
417 // Stop causes this instance of a GraphQLFrontend to cease accepting requests.
418 func (g *GraphQLFrontend) Stop() {
419 if g.IsRunning {
420 g.Server.Shutdown(context.TODO())
421 }
422 }
+152
-0
scripts/mkjson.py less more
0 #!/usr/bin/env python
1 import datetime
2 import random
3 import string
4 import json
5 import socket
6 import struct
7 import time
8 import sys
9
10 def _timezone(utc_offset):
11 '''
12 Return a string representing the timezone offset.
13
14 >>> _timezone(0)
15 '+00:00'
16 >>> _timezone(3600)
17 '+01:00'
18 >>> _timezone(-28800)
19 '-08:00'
20 >>> _timezone(-8 * 60 * 60)
21 '-08:00'
22 >>> _timezone(-30 * 60)
23 '-00:30'
24 '''
25 # Python's division uses floor(), not round() like in other languages:
26 # -1 / 2 == -1 and not -1 / 2 == 0
27 # That's why we use abs(utc_offset).
28 hours = abs(utc_offset) // 3600
29 minutes = abs(utc_offset) % 3600 // 60
30 sign = (utc_offset < 0 and '-') or '+'
31 return '%c%02d:%02d' % (sign, hours, minutes)
32
33 def _timedelta_to_seconds(td):
34 '''
35 >>> _timedelta_to_seconds(datetime.timedelta(hours=3))
36 10800
37 >>> _timedelta_to_seconds(datetime.timedelta(hours=3, minutes=15))
38 11700
39 >>> _timedelta_to_seconds(datetime.timedelta(hours=-8))
40 -28800
41 '''
42 return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10**6) / 10**6
43
44 def _utc_offset(date, use_system_timezone):
45 '''
46 Return the UTC offset of `date`. If `date` does not have any `tzinfo`, use
47 the timezone informations stored locally on the system.
48
49 >>> if time.localtime().tm_isdst:
50 ... system_timezone = -time.altzone
51 ... else:
52 ... system_timezone = -time.timezone
53 >>> _utc_offset(datetime.datetime.now(), True) == system_timezone
54 True
55 >>> _utc_offset(datetime.datetime.now(), False)
56 0
57 '''
58 if isinstance(date, datetime.datetime) and date.tzinfo is not None:
59 return _timedelta_to_seconds(date.utcoffset())
60 elif use_system_timezone:
61 if date.year < 1970:
62 # We use 1972 because 1970 doesn't have a leap day (feb 29)
63 t = time.mktime(date.replace(year=1972).timetuple())
64 else:
65 t = time.mktime(date.timetuple())
66 if time.localtime(t).tm_isdst: # pragma: no cover
67 return -time.altzone
68 else:
69 return -time.timezone
70 else:
71 return 0
72
73 def _string(d, timezone):
74 return ('%04d-%02d-%02dT%02d:%02d:%02d%s' %
75 (d.year, d.month, d.day, d.hour, d.minute, d.second, timezone))
76
77 def format(date, utc=False, use_system_timezone=True):
78 '''
79 Return a string formatted according to the :RFC:`3339`. If called with
80 `utc=True`, it normalizes `date` to the UTC date. If `date` does not have
81 any timezone information, uses the local timezone::
82
83 >>> d = datetime.datetime(2008, 4, 2, 20)
84 >>> rfc3339(d, utc=True, use_system_timezone=False)
85 '2008-04-02T20:00:00Z'
86 >>> rfc3339(d) # doctest: +ELLIPSIS
87 '2008-04-02T20:00:00...'
88
89 If called with `use_system_timezone=False` don't use the local timezone if
90 `date` does not have timezone informations and consider the offset to UTC
91 to be zero::
92
93 >>> rfc3339(d, use_system_timezone=False)
94 '2008-04-02T20:00:00+00:00'
95
96 `date` must be a `datetime.datetime`, `datetime.date` or a timestamp as
97 returned by `time.time()`::
98
99 >>> rfc3339(0, utc=True, use_system_timezone=False)
100 '1970-01-01T00:00:00Z'
101 >>> rfc3339(datetime.date(2008, 9, 6), utc=True,
102 ... use_system_timezone=False)
103 '2008-09-06T00:00:00Z'
104 >>> rfc3339(datetime.date(2008, 9, 6),
105 ... use_system_timezone=False)
106 '2008-09-06T00:00:00+00:00'
107 >>> rfc3339('foo bar')
108 Traceback (most recent call last):
109 ...
110 TypeError: Expected timestamp or date object. Got <type 'str'>.
111
112 For dates before January 1st 1970, the timezones will be the ones used in
113 1970. It might not be accurate, but on most sytem there is no timezone
114 information before 1970.
115 '''
116 # Try to convert timestamp to datetime
117 try:
118 if use_system_timezone:
119 date = datetime.datetime.fromtimestamp(date)
120 else:
121 date = datetime.datetime.utcfromtimestamp(date)
122 except TypeError:
123 pass
124
125 if not isinstance(date, datetime.date):
126 raise TypeError('Expected timestamp or date object. Got %r.' %
127 type(date))
128
129 if not isinstance(date, datetime.datetime):
130 date = datetime.datetime(*date.timetuple()[:3])
131 utc_offset = _utc_offset(date, use_system_timezone)
132 if utc:
133 # local time -> utc
134 return _string(date - datetime.timedelta(seconds=utc_offset), 'Z')
135 else:
136 return _string(date, _timezone(utc_offset))
137
138 def string2numeric_hash(text):
139 import hashlib
140 return int(hashlib.md5(text).hexdigest()[:8], 16)
141
142 entry = {}
143 entry["dns"] = {}
144 entry["timestamp_start"] = format(datetime.datetime.now())
145 entry["timestamp_end"] = format(datetime.datetime.now() + + datetime.timedelta(minutes=1))
146 for i in range(random.randint(1, 20000)):
147 rrname = ''.join(random.choice(string.ascii_lowercase) for _ in range(5))
148 rdata = socket.inet_ntoa(struct.pack('>I', string2numeric_hash(rrname)))
149 entry["dns"][(rrname + ".com")] = {"rdata": [{"rrtype": random.choice(["A", "NS", "MX"]), "rdata": rdata, "answering_host": "8.8.8.8", "count": 1, "rcode": "NOERROR"}]}
150
151 print(json.dumps(entry))