.TH nfdump 1 2005-08-19 "" ""
.SH NAME
nfdump \- netflow display and analyze program
.SH SYNOPSIS
.HP 5
.B nfdump [options] [filter]
.SH DESCRIPTION
.B nfdump
is the netflow display and analyzing program of the nfdump tool set. 
It reads the netflow data from files stored by nfcapd and processes
the flows according the options given. The filter syntax is comparable 
to tcpdump and extended for netflow data. Nfdump can also display many 
different top N flow and flow element statistics.

.SH OPTIONS
.TP 3
.B -r \fIinputfile
Read input data from \fIinputfile\fR. Default is read from stdin.
.TP 3
.B -R \fIexpr
Read input from a sequence of files in the same directory. \fIexpr\fR
may be one of:
.PD 0
.RS 4
/any/\fIdir\fR          Read recursively all files in directory \fIdir\fR.
.P
/dir/\fIfile\fR         Read all files beginning with \fIfile\fR.
.P
/dir/\fIfile1:file2\fR  Read all files from \fIfile1\fR to \fIfile2\fR.

.P
When using in combination with a sub hierarchy:
.P
/dir/\fIsub1/sub2/file1:sub3/sub4/file2\fR
.P
Read all files from \fIsub1/sub2/file1\fR 
\fIsub3/sub4/file2\fR iterating over all required hierarchy levels.

.P
Note: files are read in alphabetical sequence.
.RE
.PD
.TP 3
.B -M \fIexpr
Read input from multiple directories. \fIexpr\fR looks like:
\fI/any/path/to/dir1:dir2:dir3\fR etc. and will be expanded to the
directories: \fI/any/path/to/dir1\fR, \fI/any/path/to/dir2\fR and 
\fI/any/path/to/dir3\fR Any number of colon separated directories may 
be given. The files to read are specified by -r or -R and are expected 
to exist in all the given directories.  The options -r and -R must 
not contain any directory part when used in conjunction with -M.
.TP 3
.B -m
Sort the netflow records according the date first seen. This option is
usually only useful in conjunction with -M, when netflow records are 
read from different sources, which are not necessarily sorted.
.TP 3
.B -w \fIoutputfile
If specified writes binary netflow records to \fIoutputfile\fR ready
to be processed again with nfdump. The default output is ASCII on
stdout.
.TP 3
.B -f \fIfilterfile
Reads the filter syntax from \fIfilterfile\fR. Note: Any filter specified
directly on the command line takes precedence over -f.
.TP 3
.B -t \fItimewin
Process only flows, which fall in the time window \fItimewin\fR, where
\fItimewin\fR is YYYY/MM/dd.hh:mm:ss[-YYYY/MM/dd.hh:mm:ss]. Any parts of
the time spec may be omitted e.g YYYY/MM/dd expands to 
YYYY/MM/dd.00:00:00-YYYY/MM/dd.23:59:59 and processes all flow from a 
given day. The time window may also be specified as +/- n. In this case
it is relativ to the beginning or end of all flows. +10 means the first
10 seconds of all flows, -10 means the last 10 seconds of all flows.
.TP 3
.B -c \fInum
Limit number of records to process to the first \fInum\fR flows.
.TP 3
.B -a
Aggregate netflow data. Aggregation is done at connection level.
.TP 3
.B -A \fIfields[/netmask]
Aggregate netflow data using the specified fields, where fields is a ',' 
separated list out of \fIproto srcip dstip srcport dstport srcas dstas\fR. The default is using
all fields: \fIproto,srcip,dstip,srcport,dstport\fR. An additional netmask may be
given. In that case flows from the same subnets are aggregated. In order
to do proper aggregation, the IP version is important, for which the mask
applies. Therefore the IP protocol version must be given in the form of:
srcip4/24 for IPv4 or srcip6/64 for IPv6 address aggregation. Apply the 
protocol version for dstip respectively.
.TP 3
.B -I
Print flow statistics from file specified by -r, or timeslot specified by -R/-M. 
The printed information corresponds to pre nfdump 1.5 nfcapd stat files.
.TP 3
.B -D \fIdns
Set \fIdns\fR as nameserver to lookup hostnames.
.TP 3
.B -S
Compatibility option with pre 1.4 nfdump. Is equal to \fI-s record/packets/bytes\fR.
.TP 3
.B -s \fIstatistic[:p][/orderby]
Generate the Top N flow or flow element statistic. \fIstatistic\fR can be:
.PD 0
.RS 5
record  Statistic about arregated netflow records.
.P
srcip   Statistic about source IP addresses
.P
dstip   Statistic about destination IP addresses
.P
ip      Statistic about any (source or destination) IP addresses
.P
srcport Statistic about source ports
.P
dstport Statistic about destination ports
.P
port    Statistic about any (source or destination) ports
.P
tos     Statistic about type of service
.P
srcas   Statistic about source AS numbers
.P
dstas   Statistic about destination AS numbers
.P
as      Statistic about any (source or destination) AS numbers
.P
inif    Statistic about input interface
.P
outif   Statistic about output interface
.P
proto   Statistic about IP protocols
.RE
.RS 3
.P
By adding \fI:p\fR to the statistic name, the resulting statistic is splitted up into
transport layer protocols. Default is transport protocol independant statistics.
.P
\fIorderby\fR is optional and specifies the order by which the statistics is
ordered and can be \fIflows\fR, \fIpackets\fR, \fIbytes\fR, \fIpps\fR, \fIbps\fR 
or \fIbpp\fR. You may specify more than one \fIorderby\fR which results in the 
same statistic but ordered differently. If no \fIorderby\fR is given, statistics 
are ordered by \fIflows\fR.
You can specify as many -s flow element statistics on the command line for the 
same run. 
.P
Example: -s srcip -s ip/flows -s dstport/pps/packets/bytes -s record/bytes
.RE
.PD
.TP 3
.B -O \fIorderby
Specifies the default \fIorderby\fR for flow element statistics -s, which 
applies when no \fIorderby\fR is given at -s. \fIorderby\fR can be \fIflows\fR, 
\fIpackets\fR, \fIbytes\fR, \fIpps\fR, \fIbps\fR or \fIbpp\fR. Defaults to \fIflows\fR.
.TP 3
.B -l \fI[+/-]packet_num
Limit statistics output to those records above or below the \fIpacket_num\fR 
limit. \fIpacket_num\fR accepts positive or negative numbers followed by 'K'
, 'M' or 'G' 10E3, 10E6 or 10E9 flows respectively. See also note at -L
.TP 3
.B -L \fI[+/-]byte_num
Limit statistics output to those records above or below the \fIbyte_num\fR 
limit. \fIbyte_num\fR accepts positive or negative numbers followed by 'K'
, 'M' or 'G' 10E3, 10E6 or 10E9 bytes respectively. \fINote:\fR These limits only
apply to the statistics and aggregated outputs generated with -a -s or -S.
To filter netflow records by packets and bytes, use the filter syntax 'packets'
and 'bytes' described below.
.TP 3
.B -n \fInum
Define the number for the Top N statistics. Defaults to 10. If 0 is specified
the number is unlimited.
.TP 3
.B -o \fIformat
Selects the output format to print flows or flow record statistics (-s record). The following 
formats are available:
.PD 0
.RS 5
raw      Print each file flow record on multiple lines.
.P
line     Print each flow on one line. Default format.
.P
long     Print each flow on one line with more details
.P
extended Print each flow on one line with even more details.
.P
pipe     Machine readable format: Print all fields '|' separated.
.P
fmt:\fIformat\fR
User defined output format.
.RE
.RS 3
For each defined output format except -o fmt:<format> an IPv6 long output format exists.
\fBline6, long6 and extended6\fR. See \fIoutput formts\fR below for more information.
.RE
.PD
.TP 3
.B -K \fIkey
Anonymize all IP addresses using the CryptoPAn (Cryptography-based  
Prefix-preserving Anonymization) module. The key is used to 
initialize the Rijndael cipher. \fIkey\fR is either a 32 character 
string, or a 64 hex digit string starting with 0x. Anonymizing takes
place \fIafter\fR applying the flow filter, but before printing the flow
or writing the flow to a file.
.P
.RS 3
See http://www.cc.gatech.edu/computing/Telecomm/cryptopan/ for 
more information about CryptoPAn.
.RE
.PD
.TP 3
.B -q
Suppress the header line and the statistics at the bottom.
.TP 3
.B -N
Print the numbers in the summary line as plain numbers. Better parsing.
.TP 3
.B -v \fIfile
Verify \fIfile\fR. Print data file version, number of blocks and compression status.
.TP 3
.B -z
Compress flows. Use fast LZO1X-1 compression in output file.
.TP 3
.B -Z
Check filter syntax and exit. Sets the return value accordingly.
.TP 3
.B -X
Compiles the filer syntax and dumps the filter engine table to stdout.
This is for debugging purpose only.
.TP 3
.B -V
Print nfdump version and exit.
.TP 3
.B -h
Print help text on stdout with all options and exit.
.SH "RETURN VALUE"
Returns 
.PD 0
.RS 4 
0   No error. \fn
.P
255 Initialization failed.
.P
254 Error in filter syntax.
.P
250 Internal error.
.RE
.PD
.SH "OUTPUT FORMATS"
The output format \fBraw\fR prints each flow record on multiple lines, including
all information available in the record. This is the most detailed view on a 
flow. 
.P
Other output formats print each flow on a single line. Predefined output formats are
\fBline\fR, \fBlong\fR and \fBextended\fR
The output format \fBline\fR is the default output format when no format is specified.
It limits the imformation to the connection details as well as number of packets, 
bytes and flows.
.P
The output format \fBlong\fR is identical to the format \fBline\fR, and includes
additional information such as TCP flags and Type of Service.
.P
The output format \fBextended\fR is identical to the format \fBlong\fR, and includes
additional computed information such as \fBpps\fR, \fBbps\fR and \fBbpp\fR.
.P
\fIFields:\fR
.P
.RS 3
\fBDate flow start:\fR Start time flow first seen. ISO 8601 format 
including miliseconds.
.P
\fBDuration:\fR Duration of the flow in seconds and miliseconds. 
If flows are aggregated, \fIduration\fR is the time span over the 
entire periode of time from first seen to last seen.
.P
\fBProto:\fR Protocol used in the connection.
.P
\fBSrc IP Addr:Port:\fR Source IP address and source port.
.P
\fBDst IP Addr:Port:\fR Destination IP address and destination port.
In case of ICMP, port is decodes as type.code.
.P
\fBFlags:\fR TCP flags ORed of the connection.
.P
\fBTos:\fR Type of service.
.P
\fBPackets:\fR The number of packets in this flow. If flows are 
aggregated, the packets are summed up. 
.P
\fBBytes:\fR The number of bytes in this flow. If flows are aggregated, 
the bytes are summed up.
.P
\fBpps:\fR The calculated packets per second: number of packets / duration. 
If flows are aggregated this results in the average pps during this periode of time.
.P
\fBbps:\fR The calculated bits per second: 8 * number of bytes / duration. If flows
are aggregated this results in the average bps during this periode of time.
.P
\fBBpp:\fR The calculated bytes per packet: number of bytes / number of packets. If flows
are aggregated this results in the average bpp during this periode of time.
.P
\fBFlows:\fR Number of flows. If flows are listed only, this number is alwasy 1. If flows
are aggregated, this shows the number of aggregated flows to one record.
.RE
.PD
.P
Numbers larger than 1048576 (1024*1024), are scaled to 4 digits and one decimal digit including the
scaling factor \fBM\fR, \fBG\fR or \fBT\fR for cleaner output, e.g. \fB923.4 M\fR
.P
To make the output more readable, IPv6 addresses are shrinked down to 16 characters. The seven
most and seven least digits connected with two dots \fB'..'\fR are displayed in any normal output
formats. To display the full IPv6 address, use the appropriate long format, which is the format name
followed by a \fB6\fR. 
.P 
Example: \fB-o line\fR displays an IPv6 address as \fB2001:23..80:d01e\fR where as the format 
\fB-o line6\fR displays the IPv6 address in full length \fB2001:234:aabb::211:24ff:fe80:d01e\fR.
The combination of \fB-o line -6\fR is equivalent to \fB-o line6\fR.
.P
The \fBpipe\fR output format is intended to be read by another programm for further processing.
Values are separated by a '|'. IP addresses are printed as 4 consecutive 32bit numbers.
Output sequence:
.P
.PD 0
.RS 3
\fBAddress family\fR  PF_INET or PF_INET6
.P
\fBTime first seen\fR UNIX time seconds
.P
\fBmsec first seen\fR Mili seconds first seen
.P
\fBTime last seen\fR  UNIX time seconds
.P
\fBmsec last seen\fR  Mili seconds first seen
.P
\fBProtocol\fR        Protocol
.P
\fBSrc address\fR     Src address as 4 consecutive 32bit numbers.
.P
\fBSrc port\fR        Src port
.P
\fBDst address\fR     Dst address as 4 consecutive 32bit numbers.
.P
\fBDst port\fR        Dst port
.P
\fBSrc AS\fR          Src AS number
.P
\fBDst AS\fR          Dst AS number
.P
\fBInput IF\fR        Input Interface
.P
\fBOutput IF\fR       Output Interface
.P
\fBTCP Flags\fR       TCP Flags
.P
                      000001 FIN.
.P
                      000010 SYN
.P
                      000100 RESET
.P
                      001000 PUSH
.P
                      010000 ACK
.P
                      100000 URGENT
.P
                      e.g. 6 => SYN + RESET
.P
\fBTos\fR             Type of Service
.P
\fBPackets\fR         Packets
.P
\fBBytes\fR           Bytes
.P
.RE
.PD
.P
For IPv4 addresses only the last 32bit integer is used. All others are set to zero.
.P
The output format \fBfmt:<format>\fR allows you to define your own output format.
A format description \fBformat\fR consists of a single line containing arbitrary strings
and format specifier as described below
.P
.PD 0
.RS 3
\fB%ts\fR   Start Time - first seen
.P
\fB%te\fR   End Time - last seen
.P
\fB%td\fR   Duration
.P
\fB%pr\fR   Protocol
.P
\fB%sa\fR   Source Address
.P
\fB%da\fR   Destination Address
.P
\fB%sap\fR  Source Address:Port
.P
\fB%dap\fR  Destination Address:Port
.P
\fB%sp\fR   Source Port
.P
\fB%dp\fR   Destination Port
.P
\fB%sas\fR  Source AS
.P
\fB%das\fR  Destination AS
.P
\fB%in\fR   Input Interface num
.P
\fB%out\fR  Output Interface num
.P
\fB%pkt\fR  Packets
.P
\fB%byt\fR  Bytes
.P
\fB%fl\fR   Flows
.P
\fB%pkt\fR  Packets
.P
\fB%flg\fR  TCP Flags
.P
\fB%tos\fR  Tos
.P
\fB%bps\fR  bps - bits per second
.P
\fB%pps\fR  pps - packets per second
.P
\fB%bpp\fR  bps - Bytes per package
.RE
.PD
.P
For example the standard output format \fBlong\fR can be created as
.P
\fB-o "fmt:%ts %td %pr %sap -> %dap %flg %tos %pkt %byt %fl"\fR
.P
You may also define your own output format and have it compiled into nfdump.
See nfdump.c around line 100 for more details.
.P
.SH "FILTER"
The filter syntax is similar to the well known pcap library used by tcpdump.
The filter can be either specified on the command line after all options or 
in a separate file. It can span several lines. Anything after a '#' is treated as a 
comment and ignored to the end of the line. There is virtually no limit in 
the length of the filter expression. All keywords are case independent.
.P Syntax
Any filter consists of one or more expressions \fIexpr\fR. Any number of \fIexpr\fR
can be linked together:
.P
expr \fBand\fR expr, expr \fBor\fR expr, \fBnot\fR expr and \fB(\fR expr \fB)\fR.
.P
\fIExpr\fR can be one of the following filter primitives:
.TP 4
.I protocol version
\fBinet\fR for IPv4 and \fBinet6\fR for IPv6
.TP 4
.I protocol
\fBproto <protocol>\fR
where \fBprotocol\fR can be any known protocol such as TCP, UDP, ICMP, ICMP6 GRE, ESP, AH, or a valid protocol number.
.TP 4
.I IP address
.PD 0
.RS 4
\fI[SourceDestination]\fR \fBIP <ipaddr>\fR or
.P
\fI[SourceDestination]\fR \fBHOST <ipaddr>\fR with \fI<ipaddr>\fR as any valid IPv4, IPv6 address, or a full qualified
hostname.  In case of a hostname, the IP address is looked up in DNS. If more than a single IP address is found,
all IP addresses are chained together. ip1 or ip2 or ip3 ...
The direction tag \fISourceDestination\fR may be omitted.
.P
\fI[SourceDestination]\fR \fBIP IN\fR [\fB<iplist>\fR] 
.P
\fI[SourceDestination]\fR \fBHOST IN\fR [\fB<iplist>\fR] 
.P
\fBiplist\fR space separated list of individual \fB<ipaddr>\fR or full qualified hostnames. In case of a hostname, 
the IP address is looked up in DNS. If more than a single IP address is found, all IP addresses are put into the list.
.RE
.PD
.TP 4
.I SourceDestination
defines the IP address to be selected and can be \fBSRC\fR, 
\fBDST\fR or any combination of \fBSRC and|or DST\fR. Ommiting \fISourceDestination\fR is 
equivalent to \fBSRC or DST\fR.
.TP 4
.I inout
defines the interface to be selected and can be \fBIN\fR or
\fBOUT\fR.
.TP 4
.I network
\fI[SourceDestination]\fR \fBNET\fR \fIa.b.c.d\fR \fIm.n.r.s\fR. for IPv4 with \fIm.n.r.s\fR as netmask.
.PD 0
.RS 4
\fI[SourceDestination]\fR \fBNET\fR \fI<net>\fR / \fInum\fR with \fI<net>\fR 
as a valid IPv4 or IPv6 network and \fInum\fR as maskbits. The number of mask bits must match
the appropriate address familiy IPv4 or IPv6. Networks may be abreviated such as 172.16/16 
if they are unambiguous.
.RE
.PD
.TP 4 
.I ICMP
.PD 0
.RS 4
\fBICMP-TYPE\fR \fInum\fR, \fBICMP-CODE\fR \fInum\fR with \fInum\fR as a valid icmp type/code respectively.
This implies automatically \fBPROTO ICMP\fR
.RE
.PD
.TP 4 
.I Port
.PD 0
.RS 4
\fI[SourceDestination]\fR  \fBPORT\fR \fI[comp]\fR \fInum\fR with \fInum\fR as a valid port number.
If \fIcomp\fR is omitted, '=' is assumed.
.P
\fI[SourceDestination]\fR \fBPORT IN\fR [\fB<portlist>\fR] 
.P
\fBportlist\fR space separated list of individual port numbers
.RE
.PD
.TP 4 
.I Interface
\fI[inout]\fR  \fBIF\fR \fInum\fR with \fInum\fR as an interface number.
.TP 4
.I Flags
\fBflags\fR \fItcpflags\fR with \fItcpflags\fR as a combination of:
.PD 0
.RS 4
A    ACK.
.P
S    SYN.
.P
F    FIN.
.P
R    Reset.
.P
P    Push.
.P
U    Urgent.
.P
X    All flags on.
.RE
.PD
The ordering of the flags is not relevant. Flags not mentioned are treated as don't care.
In order to get those flows with only the SYN flag set, use the syntax '\fBflags S and not
flags AFRPU\fR'.
.TP 4 
.I TOS
Type of service: \fBtos\fR \fIvalue\fR with \fIvalue\fR 0..255.
.TP 4 
.I Packets
\fBpackets\fR \fI[comp]\fR \fInum\fR \fI[scale]\fR to specify the packet count in the netflow record.
.TP 4 
.I Bytes
\fBbytes\fR \fI[comp]\fR \fInum\fR \fI[scale]\fR to specify the byte count in the netflow record.
.TP 4 
.I Packets per second: Calculated value.
\fBpps\fR \fI[comp]\fR \fInum\fR \fI[scale]\fR to specify the pps of the flow. 
.TP 4 
.I Duration: Calculated value
\fBduration\fR \fI[comp]\fR \fInum\fR to specify the duration in miliseconds of the flow.
.TP 4 
.I Bits per second: Calculated value.
\fBbps\fR \fI[comp]\fR \fInum\fR \fI[scale]\fR to specify the bps of the flow. 
.TP 4 
.I Bytes per packet: Calculated value.
\fBbpp\fR \fI[comp]\fR \fInum\fR \fI[scale]\fR to specify the bpp of the flow. 
.TP 4
.I AS
\fI[SourceDestination]\fR  \fBAS\fR \fInum\fR with \fInum\fR as a valid AS number.
.TP 4
\fIscale\fR scaling factor. Maybe \fIk\fR \fIm\fR \fIg\fR. Factor is 1024
.TP 4
\fIcomp\fR The following comparators are supported:
.B =, ==, >, <,  EQ, LT, GT .
If \fIcomp\fR is omitted, '=' is assumed.
.SH "EXAMPLES"
.B nfdump -r /and/dir/nfcapd.200407110845 -c 100 'tcp and ( src ip 172.16.17.18 or dst ip 172.16.17.19 )'
Dumps the first 100 netflow records which match the given filter:
.P
.B nfdump -R /and/dir/nfcapd.200407110845:nfcapd.200407110945 'host 192.168.1.2'
Dumps all netflow records of host 192.168.1.2 from July 11 08:45 - 09:45
.P
.B nfdump -M /to/and/dir1:dir2 -R nfcapd.200407110845:nfcapd.200407110945  -S -n 20
Generates the Top 20 statistics from 08:45 to 09:45 from 3 sources
.P
.B nfdump -r /and/dir/nfcapd.200407110845 -S -n 20 -o extended
Generates the Top 20 statistics, extended output format
.P
.B nfdump -r /and/dir/nfcapd.200407110845 -S -n 20 'in if 5 and bps > 10k'
Generates the Top 20 statistics from flows comming from interface 5
.P
.B nfdump -r /and/dir/nfcapd.200407110845 'inet6 and tcp and ( src port > 1024 and dst port 80 )
Dumps all port 80 IPv6 connections to any web server.
.SH NOTES
Generating the statistics for data files of a few hundred MB is no problem. However
be careful if you want to create statistics of several GB of data. This may consume a lot
of memory and can take a while. Also, anonymizing IP addresses is time consuming and uses
a lot of CPU power, which reduces the number of flows per second. Therefore anonymizing
takes place only, when flow records are printed or written to files. Any internal flow
processing takes place using the original IP addresses.
.SH "SEE ALSO"
nfcapd(1), nfprofile(1), nfreplay(1)
.SH BUGS
There is still the famous last bug. Please report them - all the last bugs - back to me.