	ng_netflow 0.2.5: netgraph based traffic accounting kernel module

1. Introduction

  Netflow is a method of gathering fine traffic statistics originally
introduced in Cisco routers. It is also supported by Juniper routers,
and it is said by some other manufacturers. Netflow is de-facto standard
for gathering traffic information for billing, statistical and other
purposes. 

  There are some implementations of Netflow for *nix:
	* softflowd, http://mindrot.org/softflowd.html
	* pfflowd, http://mindrot.org/pfflowd.html
	* Fprobe, http://fprobe.sourceforge.net
	* as a part of Ntop traffic analyzing suite, http://www.ntop.org
	* ipcad, http://sf.net/projects/ipcad

These implementations (except of pfflowd) are using libpcap library
to capture traffic, and thus considered to cause heavy load on a
router. My experience with ip accounting implementations for *nix
showed me that best performance and usability is achieved by
netgraph node ng_ipacct written by Roman Palagin. So I decided to write
down netgraph node implementing Netflow. And ng_netflow's code inherits
much from ng_ipacct.

  The main goals in ng_netflow were:
	* in-kernel traffic capture and export 
	* filling in all possible fields in Netflow version 5
	  datagram

Since netgraph is currently available only on FreeBSD, ng_netflow
can be used only on FreeBSD. I have heard of NetBSD netgraph port,
but AFAIK no one have tried to use ng_netflow on NetBSD.
I hope powerful netgraph framework will be ported to OpenBSD in
future.

2. Installation

Nowadays, the correct way to install ng_netflow is to use ports:
> cd /usr/ports/net/ng_netflow && make install clean

The old way is:

> tar xzf ng_netflow-x.x.x.tar.gz
> cd ng_netflow-x.x.x
> make
> su
# make install
# kldload /modules/ng_netflow.ko

3. Stability and robustness

  ng_netflow was tested in following conditions:

   1) Production box: Duron 1 GHz, Intel Pro/100 82559 in polling.
      Collector on remote host.

      ng_netflow is connected to 2 interfaces: a 10 Mbit ethernet,
      and 2 Mbit synchronous. Average traffic is ~ 200 - 300 Kbit/s.
      The host is running BGP, it announces a /20 and accepts 2 full
      views and no default. 
      Flow cache size is about 300 entries with peak size > 2000
      entries. The host is loaded insignificantly - approximately
      97 - 98 % idle.

   2) Production box: Athlon 1.3 GHz, 3 Intel 82558 NICs + 2
      82559 NICs in polling. Collector on the same host.

      The box itself is a router in a huge LAN, and usually its
      interfaces are working at maximum speed. ng_netflow is
      connected to five 100Mbit Ethernet interfaces. Routing table
      is about 10 - 12 static entries. At the busiest hour flow
      cache grows up to 3000 flows. However, enabling netflow
      on this box did not add any noticeable load.

4. Netflow version 5 support

  In this release ng_netflow fills in all possible fields in
version 5 datagram, except of AS numbers.
If your router is running BGP, theoretically AS numbers can be filled
into export datagrams. AS paths are not stored in kernel routing table,
so we have to export them from routing daemon (GNU zebra or quagga). 
AS path can be easily added to struct rtentry(9), but the problem
is in passing AS path from userland into kernel. There are 2 ways:

 1) Altering struct rt_msghdr (see route(4)). This is easy but it means
 that new kernel will be incompatible with old userland. This is bad.
 2) Passing AS path in sockaddr. This is ugly, but does not lead to
 incompatibility.

Which one is better is a thing to discuss. Any ideas?

5. ng_ipacct + ng_netflow ?

  It is possible to run both ng_netflow and ng_ipacct on one interface.
There is nothing sophisticated in it. For example you have a
Ethernet interface with ng_ipacct on it:

         ( ng_ipacct )
     in /             \ out
    r2l \             / l2r
         (  ng_tee   )
   left /             \ right
  upper \             / lower
         ( ng_ether  )

To add ng_netflow here, you need to insert one more ng_tee into chain,
and connect netflow node on it:

( ng_netflow )               ( ng_ipacct )
iface0\                  in /             \ out
   r2l \          /     r2l \             / l2r
        (  ng_tee )----------(  ng_tee   )
  left /                                 \ right
 upper \                                 / 
        ( ng_ether )--------------------/
                             lower 

6. Cisco compatibility

  1) The export datagrams are exactly the same format as Cisco generates.
  2) The expiry algorithm tries to follow Cisco's one, but the problem is
     that Cisco's is not documented well.
  3) Output of "show" command is the same as Cisco's except of statistics
     header. The statistics are planned in next releases.
