Codebase list logswan / upstream/2.1.0
upstream/2.1.0

Tree @upstream/2.1.0 (Download .tar.gz) 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
```
                                _____
                            .xiX*****Xix.
                          .X7'        '4Xk,
                         dXl            'XX.        .
                        xXXl             XXl        .
                        4XXX             XX'
                       .  ,x            iX'   _,,xxii
                       |   ²|        ,iX7,xiiXXXXXXXl
                       |          .xi,xiXXXXXXXXXXXX:
                       .      ..iXXiXXXXXXXXXXXXXXX7.
                       .    .xXXXXXXXXXXXXXXX'XXXX7 .
                       |   ,XXXXXXXXXXXXXXXX'XXX7'  |
                       :  .XXXXX7*'"' 2XXX7'XX7'    |
  __/ \     _____    ____  \XX' _____  47'  ___  ___      _____     __
.\\_   \___/  _  \__/  _/_______\  _/______/  /  \  \____/  _  \___/  \  _____
. /     __    Y _ __   \__  _________  _____  \/\/   ____ _ _   ______ \/ __///
:/       /    |    \    |'   \/   \/    \/            \/    Y    \/   \    \  :
|\______/\_________/____|    /\____     /\_____/\_____/\____|____/\____\___/  |
+--------------------- \____/ --- \____/ ----:----------------------h7/dS!----+
                       .                     |      :
                       : .                   :      |
                       | .     Logswan       .      |
                       | :                       .  |
                       |_|_______________________|__|
                         |                       :
                                                 .
```
# Logswan

[![Build Status][1]][2]

Logswan is a fast Web log analyzer using probabilistic data structures. It is
targeted at very large log files, typically APIs logs. It has constant memory
usage regardless of the log file size, and takes approximatively 4MB of RAM.

Unique visitors counting is performed using two HyperLogLog counters (one for
IPv4, and another one for IPv6), providing a relative accuracy of 0.10%.
String representations of IP addresses are used and preferred as they offer
better precision.

Project design goals include: speed, memory-usage efficiency, and keeping the
code as simple as possible.

Logswan is **opinionated software**:

- It only supports the Common Log Format, in order to keep the parsing code
  simple. It can of course process the Combined Log Format as well (referer
  and user agent fields will be discarded)
- It does not split results per day, but log files can be split prior to
  being processed
- Input file size and bandwidth usage are reported in bytes, there are no
  plans to format or round them

## Features

Currently implemented features:

- Counting used bandwidth
- Counting number of processed lines / invalid lines
- Counting number of hits (IPv4 and IPv6 hits)
- Counting visits (unique IP addresses for both IPv4 and IPv6)
- GeoIP lookups (for both IPv4 and IPv6)
- Hourly hits distribution
- HTTP method distribution
- HTTP protocol (HTTP/1.0, HTTP/1.1, or HTTP/2.0) distribution
- HTTP status codes distribution

## Dependencies

Logswan uses the `CMake` build system and requires `Jansson` and `libmaxminddb`
libraries and header files.

## Installing dependencies

- OpenBSD: `pkg_add -r cmake jansson libmaxminddb`
- NetBSD: `pkgin in cmake jansson libmaxminddb`
- FreeBSD: `pkg install cmake jansson libmaxminddb`
- Mac OS X: `brew install cmake jansson libmaxminddb`
- Alpine Linux: `apk add cmake gcc make musl-dev jansson-dev libmaxminddb-dev`
- Debian / Ubuntu: `apt-get install build-essential cmake libjansson-dev libmaxminddb-dev`

## Building

	mkdir build
	cd build
	cmake ..
	make

Logswan has been successfully built and tested on OpenBSD, NetBSD, FreeBSD,
Mac OS X, and Linux with both Clang and GCC.

## Packages

Packages are available for the following operating systems:

- [OpenBSD][3]
- [NetBSD][4]
- [Debian][5]
- [Ubuntu][6]
- [Void Linux][7]

### GeoIP2 databases

Logswan looks for GeoIP2 databases in `${CMAKE_INSTALL_PREFIX}/share/GeoIP2` by
default, which points to `/usr/local/share/GeoIP2`.

A custom directory can be set using the `GEOIP2DIR` variable when invoking
CMake:

	cmake -DGEOIP2DIR=/var/db/GeoIP .

The free GeoLite2 databases from MaxMind can be downloaded [here][8]:

	https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz

## Usage

	logswan [-ghv] file

If file is a single dash (`-'), logswan reads from the standard input.

Options are:

	-g Enable GeoIP lookups
	-h Display usage
	-v Display version

Logswan outputs JSON data to **stdout**.

## Measuring Logswan memory usage

Heap profiling can be done using valgrind, as follows:

	valgrind --tool=massif logswan access.log
	ms_print massif.out.*

## License

Logswan is released under the BSD 2-Clause license. See `LICENSE` file for
details.

## Author

Logswan is developed by Frederic Cambus.

- Site: https://www.cambus.net

## Resources

Project homepage: https://www.logswan.org

GitHub: https://github.com/fcambus/logswan

[1]: https://api.travis-ci.org/fcambus/logswan.png?branch=master
[2]: https://travis-ci.org/fcambus/logswan
[3]: https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/www/logswan
[4]: http://pkgsrc.se/www/logswan
[5]: https://packages.debian.org/sid/logswan
[6]: https://packages.ubuntu.com/disco/logswan
[7]: https://github.com/void-linux/void-packages/tree/master/srcpkgs/logswan
[8]: https://dev.maxmind.com/geoip/geoip2/geolite2/

Commit History @upstream/2.1.0